kubernetes

[!TIP] kubernetes.io add /_print as suffix in the url, it will show pages into one page i.e.:
https://kubernetes.io/docs/setup/best-practices/
https://kubernetes.io/docs/setup/best-practices/_print/
* ramitsurana/awesome-kubernetes | Awesome Kubernetes
ibm-cloud-architecture/refarch-cloudnative-devops-kubernetes
Kubernetes Architecture Explanation | Part 1
Load Balancers vs API Gateways

core design principles

constants and well-known values and paths

`/etc/kubernetes/manifests`

[!TIP] /etc/kubernetes/manifests as the path where kubelet should look for static Pod manifests. Names of static Pod manifests are:
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

`/etc/kubernetes`

[!TIP]
important kubernetes cluster configurations /etc/kubernetes/ as the path where kubeconfig files with identities for control plane components are stored. Names of kubeconfig files are:
kubelet.conf (bootstrap-kubelet.conf during TLS bootstrap)
controller-manager.conf
scheduler.conf
admin.conf for the cluster admin and kubeadm itself

names of certificates and key files

[!TIP]
ca.crt, ca.key for the Kubernetes certificate authority
apiserver.crt, apiserver.key for the API server certificate
apiserver-kubelet-client.crt, apiserver-kubelet-client.key for the client certificate used by the API server to connect to the kubelets securely
sa.pub, sa.key for the key used by the controller manager when signing ServiceAccount
front-proxy-ca.crt, front-proxy-ca.key for the front proxy certificate authority
front-proxy-client.crt, front-proxy-client.key for the front proxy client

API server

static pod manifest

[!TIP]
apiserver-advertise-address and apiserver-bind-port to bind to; if not provided, those value defaults to the IP address of the default network interface on the machine and port 6443
service-cluster-ip-range to use for services
If an external etcd server is specified, the etcd-servers address and related TLS settings (etcd-cafile, etcd-certfile, etcd-keyfile);
if an external etcd server is not be provided, a local etcd will be used ( via host network )
If a cloud provider is specified, the corresponding --cloud-provider is configured, together with the --cloud-config path if such file exists (this is experimental, alpha and will be removed in a future version)

other api server flags

--insecure-port=0 to avoid insecure connections to the api server
--enable-bootstrap-token-auth=true to enable the BootstrapTokenAuthenticator authentication module. See TLS Bootstrapping for more details
--allow-privileged to true (required e.g. by kube proxy)
--requestheader-client-ca-file to front-proxy-ca.crt
--enable-admission-plugins to:
- NamespaceLifecycle e.g. to avoid deletion of system reserved namespaces
- LimitRanger and ResourceQuota to enforce limits on namespaces
- ServiceAccount to enforce service account automation
- PersistentVolumeLabel attaches region or zone labels to PersistentVolumes as defined by the cloud provider (This admission controller is deprecated and will be removed in a future version. It is not deployed by kubeadm by default with v1.9 onwards when not explicitly opting into using gce or aws as cloud providers)
- DefaultStorageClass to enforce default storage class on PersistentVolumeClaim objects
- DefaultTolerationSeconds
- NodeRestriction to limit what a kubelet can modify (e.g. only pods on this node)
--kubelet-preferred-address-types to InternalIP,ExternalIP,Hostname; this makes kubectl logs and other API server-kubelet communication work in environments where the hostnames of the nodes aren't resolvable
Flags for using certificates generated in previous steps:
- --client-ca-file to ca.crt
- --tls-cert-file to apiserver.crt
- --tls-private-key-file to apiserver.key
- --kubelet-client-certificate to apiserver-kubelet-client.crt
- --kubelet-client-key to apiserver-kubelet-client.key
- --service-account-key-file to sa.pub
- --requestheader-client-ca-file to front-proxy-ca.crt
- --proxy-client-cert-file to front-proxy-client.crt
- --proxy-client-key-file to front-proxy-client.key
Other flags for securing the front proxy (API Aggregation) communications:
- --requestheader-username-headers=X-Remote-User
- --requestheader-group-headers=X-Remote-Group
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-allowed-names=front-proxy-client

controller manager

static Pod manifest

[!TIP]
If kubeadm is invoked specifying a --pod-network-cidr, the subnet manager feature required for some CNI network plugins is enabled by setting:
--allocate-node-cidrs=true
--cluster-cidr and --node-cidr-mask-size flags according to the given CIDR
If a cloud provider is specified, the corresponding --cloud-provider is specified, together with the --cloud-config path if such configuration file exists (this is experimental, alpha and will be removed in a future version)

other flags

--controllers enabling all the default controllers plus BootstrapSigner and TokenCleaner controllers for TLS bootstrap. See TLS Bootstrapping for more details
--use-service-account-credentials to true
Flags for using certificates generated in previous steps:
- --root-ca-file to ca.crt
- --cluster-signing-cert-file to ca.crt, if External CA mode is disabled, otherwise to ""
- --cluster-signing-key-file to ca.key, if External CA mode is disabled, otherwise to ""
- --service-account-private-key-file to sa.key

flow

pod creation

ingress traffic

ports and protocols

contol plane

PROTOCOL

DIRECTION

PORT RANGE

PURPOSE

USED BY

TCP

Inbound

6443

Kubernetes API server

All

TCP

Inbound

2379-2380

etcd server client API

kube-apiserver, etcd

TCP

Inbound

10250

Kubelet API

Self, Control plane

TCP

Inbound

10259

kube-scheduler

Self

TCP

Inbound

10257

kube-controller-manager

Self

worker node(s)

PROTOCOL

DIRECTION

PORT RANGE

PURPOSE

USED BY

TCP

Inbound

10250

Kubelet API

Self, Control plane

TCP

Inbound

30000-32767

NodePort Services

All

architecture

control pannel

kube-apiserver

etcd

kube-scheduler

controller manager

ccm : cloud controller manager

work node

[!NOTE]
supported endpoints
linux

RUNTIME

PATH TO UNIX DOMAIN SOCKET

containerd

unix:///var/run/containerd/containerd.sock

CRI-O

unix:///var/run/crio/crio.sock

Docker Engine (using cri-dockerd)

unix:///var/run/cri-dockerd.sock

windows

RUNTIME

PATH TO UNIX DOMAIN SOCKET

containerd

npipe:////./pipe/containerd-containerd

Docker Engine (using cri-dockerd)

npipe:////./pipe/cri-dockerd

kubelet

kube proxy

cri-o : container runtime

jsonpath

[!NOTE|label:references:]
JSONPath Support
how to make kubectl jsonpath output on separate lines
JSONPath Reference
Basic JSONPath Rules
Playing with kubectl output

options

explain

$ kubectl explain hpa
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

DESCRIPTION:
     configuration of a horizontal pod autoscaler.

FIELDS:
   apiVersion <string>
   ...

$ kubectl explain configmap
KIND:     ConfigMap
VERSION:  v1

DESCRIPTION:
     ConfigMap holds configuration data for pods to consume.

FIELDS:
   apiVersion <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
     ...

kubectl alias

`__start_kubectl`

$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ cat >> ~/.bashrc <<EOF
alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
for _i in k kc ki kk; do complete -F __start_kubectl "${_i}"; done
EOF
$ source ~/.bashrc

`_complete_alias`

$ sudo dnf install -y bash-completion

# download bash_completion.sh for kubectl
$ curl -fsSL https://github.com/cykerway/complete-alias/raw/master/complete_alias -o ~/.bash_completion.sh
# or rhel/centos
$ sudo curl -fsSL https://github.com/marslo/dotfiles/raw/main/.marslo/.completion/complete_alias -o /etc/profile.d/complete_alias.sh
$ sudo chmod +x !$

$ cat >> ~/.bashrc << EOF
command -v kubectl >/dev/null && source <(kubectl completion bash)
test -f ~/.bash_completion.sh && source ~/.bash_completion.sh
# or
# test -f /etc/profile.d/complete_alias.sh && source /etc/profile.d/complete_alias.sh

alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
alias km='kubectl -n monitoring'

complete -o default -F __start_kubectl kubecolor
complete -o nosort -o bashdefault -o default -F _complete_alias $(alias | sed -rn 's/^alias ([^=]+)=.+kubec.+$/\1/p' | xargs)
EOF

$ source ~/.bashrc

kubecolor

$ [[ -d /tmp/kubecolor ]] && sudo mkdir -p /tmp/kubecolor
$ curl -fsSL https://github.com/hidetatz/kubecolor/releases/download/v0.0.25/kubecolor_0.0.25_Linux_x86_64.tar.gz | tar xzf - -C /tmp/kubecolor
$ sudo mv /tmp/kubecolor/kubecolor /usr/local/bin/
$ sudo chmod +x /usr/local/bin/kubecolor

token

check token

$ sudo kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
bop765.brol9nsrw820gmbi   <forever>   <never>                     authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
khhfwa.jvkvrpiknx4o6ffy   19h         2018-07-13T11:37:43+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

generate token

[!NOET|label:see also:]
* iMarslo: get join command
* iMarslo: retrive join command

$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.1.100:6443 --token lhb1ln.oj0fqwgd1yl7l9xp --discovery-token-ca-cert-hash sha256:cba8df87dcb70c83c19af72c02e4886fcc7b0cf05319084751e6ece688443bde

$ sudo kubeadm token create --print-join-command --ttl=0
kubeadm join 192.168.1.100:6443 --token bop765.brol9nsrw820gmbi --discovery-token-ca-cert-hash sha256:c8650c56faf72b8bf71c576f0d13f44c93bea2d21d4329c64bb97cba439af5c3

tear down

[!TIP]
How to completely uninstall kubernetes
* iMarslo: kubeadm reset and teardown

ubuntu

$ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
$ kubectl delete node <node name>

$ sudo kubeadm  reset
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

$ systemctl stop kubelet
$ docker system prune -a -f
$ systemctl stop docker

$ sudo rm -rf /etc/kubernetes/
$ sudo rm -rf /var/lib/cni/
$ sudo rm -rf /var/lib/kubelet/*
$ sudo rm -rf /etc/cni/
$ sudo ifconfig cni0 down
$ sudo ifconfig flannel.1 down

$ rm -rf ~/.kube/

$ sudo apt purge kubeadm kubectl kubelet kubernetes-cni kube*
$ sudo apt autoremove

CentOS/RHEL

$ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
$ kubectl delete node <node name>

$ sudo kubeadm reset -f --v=5
$ docker system prune -a -f

# stop and disable services
$ systemctl stop kubelet
$ systemctl disable kubelet
$ systemctl stop docker
$ systemctl disable docker
$ systemctl stop crio
# or
$ systemctl disable crio
$ sudo rm -rf /etc/systemd/system/multi-user.target.wants/kubelet.service
$ sudo rm -rf /etc/systemd/system/multi-user.target.wants/docker.service
$ sudo rm -rf /usr/lib/systemd/system/docker.service
$ sudo rm -rf /usr/lib/systemd/system/kubelet.service.d/

# network interface
$ sudo ifconfig cni0 down
$ sudo ip link delete cni0
$ sudo ifconfig flannel.1 down
$ sudo ip link delete flannel.1
$ sudo ifconfig docker0 down
$ sudo ip link delete docker0
$ sudo ifconfig vxlan.calico down
$ sudo ip link delete vxlan.calico

$ sudo yum versionlock delete docker-ce
$ sudo yum versionlock delete docker-ce-cli
$ sudo yum versionlock delete kubeadm
$ sudo yum versionlock delete kubelet
$ sudo yum versionlock delete kubectl
$ sudo yum versionlock delete kubernetes-cni
# or
$ sudo yum versionlock clear
$ sudo yum remove -y docker-ce docker-ce-cli containerd.io kubectl kubeadm kubelet kubernetes-cni
$ sudo yum autormeove

$ sudo rm -rf /etc/cni /etc/kubernetes /etc/docker $HOME/.kube
$ sudo rm -rf /usr/libexec/docker /usr/libexec/kubernetes

$ sudo rm -rf /var/lib/etcd/               # optional
$ sudo rm -rf /var/lib/kubelet/ /var/lib/dockershim /var/lib/yum/repos/x86_64/7/kubernetes /var/log/pods /var/log/containers
$ sudo rm -rf /var/run/docker.sock
$ sudo rm -rf /var/cache/yum/x86_64/7/kubernetes

$ sudo yum clean all
$ sudo rm -rf /var/cache/yum
$ sudo yum makecache
$ sudo yum check-update

references

references:
* Play with Kubernetes
* Glossary
* Install Kubernetes Cluster on Centos 8 With Kubeadm and CRI-O
* 在 CentOS 上部署 Kubernetes 集群
* Kubernetes 中文指南/云原生应用架构实战手册
* Create static Pods
* Implementation details
* Scheduling, Preemption and Eviction
* Resource Management for Pods and Containers
* Administer a Cluster
* Kubernetes概念辨析
* Step by step installation of 3-nodes Kubernetes Cluster on CentOS 8
* KUBERNETES INSTALLATION ON CENTOS 8
* Reference > Configuration APIs
KUBERNETES: AN OVERVIEW
What Is Kubernetes: A Container Orchestration Platform
KUBERNETES, OPEN-SOURCE CONTAINER ORCHESTRATION TECHNOLOGY
Boosting your kubectl productivity
23 Advanced kubectl commands
8 Kubernetes Tips and Tricks
Linux namespace简介
Well-Known Labels, Annotations and Taints
* best practices
12 Kubernetes Configuration Best Practices
Best Kubernetes Certifications for 2023 [Ranked]
Important Kubernetes Cluster Configurations
Kubernetes Architecture
Kubernetes Architecture Explained [Comprehensive Guide]
* How to Access Kubernetes API Server
Kubernetes（k8s）基础概念介绍
Migrating a cluster from flannel to Calico
Kubernetes 网络组件
resources:
Free Kubernetes Ebook: Kubernetes up and running
How to Learn Kubernetes (Complete Roadmap & Resources)
kelseyhightower/kubernetes-the-hard-way
* Kubernetes Tutorials For Beginners: 43 Comprehensive Guides
* How to Setup Jenkins Build Agents on Kubernetes Pods
Kubernetes Learning Resources
Kubernetes Documentation/Tasks

Previousvirtualization Nextinit

Last updated 1 year ago

Was this helpful?

hashtag/etc/kubernetes/manifests

hashtag/etc/kubernetes

hashtagnames of certificates and key files

hashtagother api server flags

hashtagother flags

hashtagflow

hashtagpod creation

hashtagingress traffic

hashtagcontol plane

hashtagworker node(s)

hashtagcontrol pannel

hashtagkube-apiserver

hashtagetcd

hashtagkube-scheduler

hashtagcontroller manager

hashtagccm : cloud controller manager

hashtagwork node

hashtagkubelet

hashtagkube proxy

hashtagcri-o : container runtime

hashtagjsonpath

hashtagoptions

hashtagexplain

hashtag__start_kubectl

hashtag_complete_alias

hashtagkubecolor

hashtagtoken

hashtagcheck token

hashtaggenerate token

hashtagreferences

`/etc/kubernetes/manifests`

`/etc/kubernetes`

names of certificates and key files

other api server flags

other flags

flow

pod creation

ingress traffic

contol plane

worker node(s)

control pannel

kube-apiserver

etcd

kube-scheduler

controller manager

ccm : cloud controller manager

work node

kubelet

kube proxy

cri-o : container runtime

jsonpath

options

explain

`__start_kubectl`

`_complete_alias`

kubecolor

token

check token

generate token

references