kubernetes

• core design principles

  • constants and well-known values and paths

    • /etc/kubernetes/manifests

    • /etc/kubernetes

    • names of certificates and key files

  • API server

    • static pod manifest

    • other api server flags

  • controller manager

    • static Pod manifest

    • other flags

  • flow

    • pod creation

    • ingress traffic

• ports and protocols

  • contol plane

  • worker node(s)

• architecture

  • control pannel

    • kube-apiserver

    • etcd

    • kube-scheduler

    • controller manager

    • ccm : cloud controller manager

  • work node

    • kubelet

    • kube proxy

    • cri-o : container runtime

• jsonpath

• options

  • explain

• kubectl alias

  • __start_kubectl

  • _complete_alias

  • kubecolor

• token

  • check token

  • generate token

• tear down

• references

[!TIP] kubernetes.io add /_print as suffix in the url, it will show pages into one page i.e.:

• https://kubernetes.io/docs/setup/best-practices/
• https://kubernetes.io/docs/setup/best-practices/_print/
• * ramitsurana/awesome-kubernetes | Awesome Kubernetes
• ibm-cloud-architecture/refarch-cloudnative-devops-kubernetes
• Kubernetes Architecture Explanation | Part 1
• Load Balancers vs API Gateways

kubernetes orchestration control panel

kubernetes technology

core design principles

constants and well-known values and paths

/etc/kubernetes/manifests

[!TIP] /etc/kubernetes/manifests as the path where kubelet should look for static Pod manifests. Names of static Pod manifests are:

• etcd.yaml

• kube-apiserver.yaml

• kube-controller-manager.yaml

• kube-scheduler.yaml

/etc/kubernetes

[!TIP]

• important kubernetes cluster configurations /etc/kubernetes/ as the path where kubeconfig files with identities for control plane components are stored. Names of kubeconfig files are:
• kubelet.conf (bootstrap-kubelet.conf during TLS bootstrap)

• controller-manager.conf

• scheduler.conf

• admin.conf for the cluster admin and kubeadm itself

names of certificates and key files

[!TIP]

• ca.crt, ca.key for the Kubernetes certificate authority

• apiserver.crt, apiserver.key for the API server certificate

• apiserver-kubelet-client.crt, apiserver-kubelet-client.key for the client certificate used by the API server to connect to the kubelets securely

• sa.pub, sa.key for the key used by the controller manager when signing ServiceAccount

• front-proxy-ca.crt, front-proxy-ca.key for the front proxy certificate authority

• front-proxy-client.crt, front-proxy-client.key for the front proxy client

API server

static pod manifest

[!TIP]

• apiserver-advertise-address and apiserver-bind-port to bind to; if not provided, those value defaults to the IP address of the default network interface on the machine and port 6443

• service-cluster-ip-range to use for services

• If an external etcd server is specified, the etcd-servers address and related TLS settings (etcd-cafile, etcd-certfile, etcd-keyfile);

  • if an external etcd server is not be provided, a local etcd will be used ( via host network )

• If a cloud provider is specified, the corresponding --cloud-provider is configured, together with the --cloud-config path if such file exists (this is experimental, alpha and will be removed in a future version)

other api server flags

• --insecure-port=0 to avoid insecure connections to the api server

• --enable-bootstrap-token-auth=true to enable the BootstrapTokenAuthenticator authentication module. See TLS Bootstrapping for more details

• --allow-privileged to true (required e.g. by kube proxy)

• --requestheader-client-ca-file to front-proxy-ca.crt

• --enable-admission-plugins to:

  • NamespaceLifecycle e.g. to avoid deletion of system reserved namespaces

  • LimitRanger and ResourceQuota to enforce limits on namespaces

  • ServiceAccount to enforce service account automation

  • PersistentVolumeLabel attaches region or zone labels to PersistentVolumes as defined by the cloud provider (This admission controller is deprecated and will be removed in a future version. It is not deployed by kubeadm by default with v1.9 onwards when not explicitly opting into using gce or aws as cloud providers)

  • DefaultStorageClass to enforce default storage class on PersistentVolumeClaim objects

  • DefaultTolerationSeconds

  • NodeRestriction to limit what a kubelet can modify (e.g. only pods on this node)

• --kubelet-preferred-address-types to InternalIP,ExternalIP,Hostname; this makes kubectl logs and other API server-kubelet communication work in environments where the hostnames of the nodes aren't resolvable

• Flags for using certificates generated in previous steps:

  • --client-ca-file to ca.crt

  • --tls-cert-file to apiserver.crt

  • --tls-private-key-file to apiserver.key

  • --kubelet-client-certificate to apiserver-kubelet-client.crt

  • --kubelet-client-key to apiserver-kubelet-client.key

  • --service-account-key-file to sa.pub

  • --requestheader-client-ca-file to front-proxy-ca.crt

  • --proxy-client-cert-file to front-proxy-client.crt

  • --proxy-client-key-file to front-proxy-client.key

• Other flags for securing the front proxy (API Aggregation) communications:

  • --requestheader-username-headers=X-Remote-User

  • --requestheader-group-headers=X-Remote-Group

  • --requestheader-extra-headers-prefix=X-Remote-Extra-

  • --requestheader-allowed-names=front-proxy-client

controller manager

static Pod manifest

[!TIP]

• If kubeadm is invoked specifying a --pod-network-cidr, the subnet manager feature required for some CNI network plugins is enabled by setting:

  • --allocate-node-cidrs=true

  • --cluster-cidr and --node-cidr-mask-size flags according to the given CIDR

• If a cloud provider is specified, the corresponding --cloud-provider is specified, together with the --cloud-config path if such configuration file exists (this is experimental, alpha and will be removed in a future version)

other flags

• --controllers enabling all the default controllers plus BootstrapSigner and TokenCleaner controllers for TLS bootstrap. See TLS Bootstrapping for more details

• --use-service-account-credentials to true

• Flags for using certificates generated in previous steps:

  • --root-ca-file to ca.crt

  • --cluster-signing-cert-file to ca.crt, if External CA mode is disabled, otherwise to ""

  • --cluster-signing-key-file to ca.key, if External CA mode is disabled, otherwise to ""

  • --service-account-private-key-file to sa.key

flow

pod creation

ingress traffic

ingress traffic flow

ports and protocols

contol plane

PROTOCOL	DIRECTION	PORT RANGE	PURPOSE	USED BY
TCP	Inbound	6443	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10259	kube-scheduler	Self
TCP	Inbound	10257	kube-controller-manager	Self

worker node(s)

PROTOCOL	DIRECTION	PORT RANGE	PURPOSE	USED BY
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	30000-32767	NodePort Services	All

architecture

Kubernetes Architecture

control pannel

kube-apiserver

Kubernetes Architecture : kube-apiserver

etcd

Kubernetes Architecture : etcd

kube-scheduler

Kubernetes Architecture : kube-scheduler

controller manager

Kubernetes Architecture : kube conntroller manager

ccm : cloud controller manager

Kubernetes Architecture : ccm

work node

[!NOTE]

• supported endpoints
• linux

RUNTIME	PATH TO UNIX DOMAIN SOCKET
containerd	unix:///var/run/containerd/containerd.sock
CRI-O	unix:///var/run/crio/crio.sock
Docker Engine (using cri-dockerd)	unix:///var/run/cri-dockerd.sock

• windows

RUNTIME	PATH TO UNIX DOMAIN SOCKET
containerd	npipe:////./pipe/containerd-containerd
Docker Engine (using cri-dockerd)	npipe:////./pipe/cri-dockerd

kubelet

Kubernetes Architecture : kubelet

kube proxy

Kubernetes Architecture : kube-proxy

cri-o : container runtime

Kubernetes Architecture : cri-o

jsonpath

[!NOTE|label:references:]

• JSONPath Support
• how to make kubectl jsonpath output on separate lines
• JSONPath Reference
• Basic JSONPath Rules
• Playing with kubectl output

options

explain

$ kubectl explain hpa
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

DESCRIPTION:
     configuration of a horizontal pod autoscaler.

FIELDS:
   apiVersion <string>
   ...

• or

  $ kubectl explain configmap
  KIND:     ConfigMap
  VERSION:  v1
  
  DESCRIPTION:
       ConfigMap holds configuration data for pods to consume.
  
  FIELDS:
     apiVersion <string>
       APIVersion defines the versioned schema of this representation of an
       object. Servers should convert recognized schemas to the latest internal
       value, and may reject unrecognized values. More info:
       https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
       ...

kubectl alias

__start_kubectl

$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ cat >> ~/.bashrc <<EOF
alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
for _i in k kc ki kk; do complete -F __start_kubectl "${_i}"; done
EOF
$ source ~/.bashrc

_complete_alias

$ sudo dnf install -y bash-completion

# download bash_completion.sh for kubectl
$ curl -fsSL https://github.com/cykerway/complete-alias/raw/master/complete_alias -o ~/.bash_completion.sh
# or rhel/centos
$ sudo curl -fsSL https://github.com/marslo/dotfiles/raw/main/.marslo/.completion/complete_alias -o /etc/profile.d/complete_alias.sh
$ sudo chmod +x !$

$ cat >> ~/.bashrc << EOF
command -v kubectl >/dev/null && source <(kubectl completion bash)
test -f ~/.bash_completion.sh && source ~/.bash_completion.sh
# or
# test -f /etc/profile.d/complete_alias.sh && source /etc/profile.d/complete_alias.sh

alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
alias km='kubectl -n monitoring'

complete -o default -F __start_kubectl kubecolor
complete -o nosort -o bashdefault -o default -F _complete_alias $(alias | sed -rn 's/^alias ([^=]+)=.+kubec.+$/\1/p' | xargs)
EOF

$ source ~/.bashrc

kubecolor

$ [[ -d /tmp/kubecolor ]] && sudo mkdir -p /tmp/kubecolor
$ curl -fsSL https://github.com/hidetatz/kubecolor/releases/download/v0.0.25/kubecolor_0.0.25_Linux_x86_64.tar.gz | tar xzf - -C /tmp/kubecolor
$ sudo mv /tmp/kubecolor/kubecolor /usr/local/bin/
$ sudo chmod +x /usr/local/bin/kubecolor

token

check token

$ sudo kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
bop765.brol9nsrw820gmbi   <forever>   <never>                     authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
khhfwa.jvkvrpiknx4o6ffy   19h         2018-07-13T11:37:43+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

generate token

[!NOET|label:see also:]

• * iMarslo: get join command
• * iMarslo: retrive join command

$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.1.100:6443 --token lhb1ln.oj0fqwgd1yl7l9xp --discovery-token-ca-cert-hash sha256:cba8df87dcb70c83c19af72c02e4886fcc7b0cf05319084751e6ece688443bde

$ sudo kubeadm token create --print-join-command --ttl=0
kubeadm join 192.168.1.100:6443 --token bop765.brol9nsrw820gmbi --discovery-token-ca-cert-hash sha256:c8650c56faf72b8bf71c576f0d13f44c93bea2d21d4329c64bb97cba439af5c3

tear down

[!TIP]

• How to completely uninstall kubernetes
• * iMarslo: kubeadm reset and teardown

• ubuntu

  $ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  $ kubectl delete node <node name>
  
  $ sudo kubeadm  reset
  [preflight] Running pre-flight checks.
  [reset] Stopping the kubelet service.
  [reset] Unmounting mounted directories in "/var/lib/kubelet"
  [reset] Removing kubernetes-managed containers.
  [reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
  [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
  [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
  
  $ systemctl stop kubelet
  $ docker system prune -a -f
  $ systemctl stop docker
  
  $ sudo rm -rf /etc/kubernetes/
  $ sudo rm -rf /var/lib/cni/
  $ sudo rm -rf /var/lib/kubelet/*
  $ sudo rm -rf /etc/cni/
  $ sudo ifconfig cni0 down
  $ sudo ifconfig flannel.1 down
  
  $ rm -rf ~/.kube/
  
  $ sudo apt purge kubeadm kubectl kubelet kubernetes-cni kube*
  $ sudo apt autoremove

• CentOS/RHEL

  $ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  $ kubectl delete node <node name>
  
  $ sudo kubeadm reset -f --v=5
  $ docker system prune -a -f
  
  # stop and disable services
  $ systemctl stop kubelet
  $ systemctl disable kubelet
  $ systemctl stop docker
  $ systemctl disable docker
  $ systemctl stop crio
  # or
  $ systemctl disable crio
  $ sudo rm -rf /etc/systemd/system/multi-user.target.wants/kubelet.service
  $ sudo rm -rf /etc/systemd/system/multi-user.target.wants/docker.service
  $ sudo rm -rf /usr/lib/systemd/system/docker.service
  $ sudo rm -rf /usr/lib/systemd/system/kubelet.service.d/
  
  # network interface
  $ sudo ifconfig cni0 down
  $ sudo ip link delete cni0
  $ sudo ifconfig flannel.1 down
  $ sudo ip link delete flannel.1
  $ sudo ifconfig docker0 down
  $ sudo ip link delete docker0
  $ sudo ifconfig vxlan.calico down
  $ sudo ip link delete vxlan.calico
  
  $ sudo yum versionlock delete docker-ce
  $ sudo yum versionlock delete docker-ce-cli
  $ sudo yum versionlock delete kubeadm
  $ sudo yum versionlock delete kubelet
  $ sudo yum versionlock delete kubectl
  $ sudo yum versionlock delete kubernetes-cni
  # or
  $ sudo yum versionlock clear
  $ sudo yum remove -y docker-ce docker-ce-cli containerd.io kubectl kubeadm kubelet kubernetes-cni
  $ sudo yum autormeove
  
  $ sudo rm -rf /etc/cni /etc/kubernetes /etc/docker $HOME/.kube
  $ sudo rm -rf /usr/libexec/docker /usr/libexec/kubernetes
  
  $ sudo rm -rf /var/lib/etcd/               # optional
  $ sudo rm -rf /var/lib/kubelet/ /var/lib/dockershim /var/lib/yum/repos/x86_64/7/kubernetes /var/log/pods /var/log/containers
  $ sudo rm -rf /var/run/docker.sock
  $ sudo rm -rf /var/cache/yum/x86_64/7/kubernetes
  
  $ sudo yum clean all
  $ sudo rm -rf /var/cache/yum
  $ sudo yum makecache
  $ sudo yum check-update

references

• references:

  • * Play with Kubernetes
  • * Glossary
  • * Install Kubernetes Cluster on Centos 8 With Kubeadm and CRI-O
  • * 在 CentOS 上部署 Kubernetes 集群
  • * Kubernetes 中文指南/云原生应用架构实战手册
  • * Create static Pods
  • * Implementation details
  • * Scheduling, Preemption and Eviction
  • * Resource Management for Pods and Containers
  • * Administer a Cluster
  • * Kubernetes概念辨析
  • * Step by step installation of 3-nodes Kubernetes Cluster on CentOS 8
  • * KUBERNETES INSTALLATION ON CENTOS 8
  • * Reference > Configuration APIs
  • KUBERNETES: AN OVERVIEW
  • What Is Kubernetes: A Container Orchestration Platform
  • KUBERNETES, OPEN-SOURCE CONTAINER ORCHESTRATION TECHNOLOGY
  • Boosting your kubectl productivity
  • 23 Advanced kubectl commands
  • 8 Kubernetes Tips and Tricks
  • Linux namespace简介
  • Well-Known Labels, Annotations and Taints
  • * best practices
  • 12 Kubernetes Configuration Best Practices
  • Best Kubernetes Certifications for 2023 [Ranked]
  • Important Kubernetes Cluster Configurations
  • Kubernetes Architecture

    • Kubernetes Architecture Explained [Comprehensive Guide]
  • * How to Access Kubernetes API Server
  • Kubernetes（k8s）基础概念介绍
  • Migrating a cluster from flannel to Calico
  • Kubernetes 网络组件
• resources:

  • Free Kubernetes Ebook: Kubernetes up and running
  • How to Learn Kubernetes (Complete Roadmap & Resources)
  • kelseyhightower/kubernetes-the-hard-way
  • * Kubernetes Tutorials For Beginners: 43 Comprehensive Guides
  • * How to Setup Jenkins Build Agents on Kubernetes Pods
  • Kubernetes Learning Resources
  • Kubernetes Documentation/Tasks