# kubernetes
* [core design principles](#core-design-principles)
* [constants and well-known values and paths](#constants-and-well-known-values-and-paths)
* [`/etc/kubernetes/manifests`](#etckubernetesmanifests)
* [`/etc/kubernetes`](#etckubernetes)
* [names of certificates and key files](#names-of-certificates-and-key-files)
* [API server](#api-server)
* [static pod manifest](#static-pod-manifest)
* [other api server flags](#other-api-server-flags)
* [controller manager](#controller-manager)
* [static Pod manifest](#static-pod-manifest)
* [other flags](#other-flags)
* [flow](#flow)
* [pod creation](#pod-creation)
* [ingress traffic](#ingress-traffic)
* [ports and protocols](#ports-and-protocols)
* [contol plane](#contol-plane)
* [worker node(s)](#worker-nodes)
* [architecture](#architecture)
* [control pannel](#control-pannel)
* [kube-apiserver](#kube-apiserver)
* [etcd](#etcd)
* [kube-scheduler](#kube-scheduler)
* [controller manager](#controller-manager-1)
* [ccm : cloud controller manager](#ccm--cloud-controller-manager)
* [work node](#work-node)
* [kubelet](#kubelet)
* [kube proxy](#kube-proxy)
* [cri-o : container runtime](#cri-o--container-runtime)
* [jsonpath](#jsonpath)
* [options](#options)
* [explain](#explain)
* [kubectl alias](#kubectl-alias)
* [`__start_kubectl`](#__start_kubectl)
* [`_complete_alias`](#_complete_alias)
* [kubecolor](#kubecolor)
* [token](#token)
* [check token](#check-token)
* [generate token](#generate-token)
* [tear down](#tear-down)
* [references](#references)
> \[!TIP] `kubernetes.io` add `/_print` as suffix in the url, it will show pages into one page i.e.:
>
> *
> *
> * [\* ramitsurana/awesome-kubernetes](https://github.com/ramitsurana/awesome-kubernetes) | [Awesome Kubernetes](https://ramitsurana.github.io/awesome-kubernetes/)
> * [ibm-cloud-architecture/refarch-cloudnative-devops-kubernetes](https://github.com/ibm-cloud-architecture/refarch-cloudnative-devops-kubernetes)
> * [Kubernetes Architecture Explanation | Part 1](https://dev.to/mrcaption49/kubernetes-architecture-part-1-12f7)
> * [Load Balancers vs API Gateways](https://vaibhav342.hashnode.dev/load-balancers-vs-api-gateways)
## [core design principles](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#core-design-principles)
### [constants and well-known values and paths](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#constants-and-well-known-values-and-paths)
#### `/etc/kubernetes/manifests`
> \[!TIP] `/etc/kubernetes/manifests` as the path where kubelet should look for static Pod manifests. Names of static Pod manifests are:
>
> * `etcd.yaml`
> * `kube-apiserver.yaml`
> * `kube-controller-manager.yaml`
> * `kube-scheduler.yaml`
#### `/etc/kubernetes`
> \[!TIP]
>
> * [important kubernetes cluster configurations](https://devopscube.com/kubernetes-cluster-configurations/) `/etc/kubernetes/` as the path where kubeconfig files with identities for control plane components are stored. Names of kubeconfig files are:
> * `kubelet.conf` (bootstrap-kubelet.conf during TLS bootstrap)
> * `controller-manager.conf`
> * `scheduler.conf`
> * `admin.conf` for the cluster admin and kubeadm itself
#### names of certificates and key files
> \[!TIP]
>
> * `ca.crt`, `ca.key` for the Kubernetes certificate authority
> * `apiserver.crt`, `apiserver.key` for the API server certificate
> * `apiserver-kubelet-client.crt`, `apiserver-kubelet-client.key` for the client certificate used by the API server to connect to the kubelets securely
> * `sa.pub`, `sa.key` for the key used by the controller manager when signing ServiceAccount
> * `front-proxy-ca.crt`, `front-proxy-ca.key` for the front proxy certificate authority
> * `front-proxy-client.crt`, `front-proxy-client.key` for the front proxy client
### [API server](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#api-server)
#### [static pod manifest](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/)
> \[!TIP]
>
> * `apiserver-advertise-address` and `apiserver-bind-port` to bind to; if not provided, those value defaults to the IP address of the default network interface on the machine and port `6443`
> * `service-cluster-ip-range` to use for services
> * If an external etcd server is specified, the `etcd-servers` address and related TLS settings (`etcd-cafile`, `etcd-certfile`, `etcd-keyfile`);
> * if an external etcd server is not be provided, a local etcd will be used ( via host network )
> * If a cloud provider is specified, the corresponding `--cloud-provider` is configured, together with the `--cloud-config` path if such file exists (this is experimental, alpha and will be removed in a future version)
#### other api server flags
* `--insecure-port=0` to avoid insecure connections to the api server
* `--enable-bootstrap-token-auth=true` to enable the BootstrapTokenAuthenticator authentication module. See TLS Bootstrapping for more details
* `--allow-privileged` to true (required e.g. by kube proxy)
* `--requestheader-client-ca-file` to front-proxy-ca.crt
* `--enable-admission-plugins` to:
* `NamespaceLifecycle` e.g. to avoid deletion of system reserved namespaces
* `LimitRanger` and `ResourceQuota` to enforce limits on namespaces
* `ServiceAccount` to enforce service account automation
* `PersistentVolumeLabel` attaches region or zone labels to PersistentVolumes as defined by the cloud provider (This admission controller is deprecated and will be removed in a future version. It is not deployed by kubeadm by default with v1.9 onwards when not explicitly opting into using gce or aws as cloud providers)
* `DefaultStorageClass` to enforce default storage class on PersistentVolumeClaim objects
* `DefaultTolerationSeconds`
* `NodeRestriction` to limit what a kubelet can modify (e.g. only pods on this node)
* `--kubelet-preferred-address-types` to `InternalIP`,`ExternalIP`,`Hostname`; this makes `kubectl logs` and other API server-kubelet communication work in environments where the hostnames of the nodes aren't resolvable
* Flags for using certificates generated in previous steps:
* `--client-ca-file` to `ca.crt`
* `--tls-cert-file` to `apiserver.crt`
* `--tls-private-key-file` to `apiserver.key`
* `--kubelet-client-certificate` to `apiserver-kubelet-client.crt`
* `--kubelet-client-key` to `apiserver-kubelet-client.key`
* `--service-account-key-file` to `sa.pub`
* `--requestheader-client-ca-file` to `front-proxy-ca.crt`
* `--proxy-client-cert-file` to `front-proxy-client.crt`
* `--proxy-client-key-file` to `front-proxy-client.key`
* Other flags for securing the front proxy (API Aggregation) communications:
* `--requestheader-username-headers=X-Remote-User`
* `--requestheader-group-headers=X-Remote-Group`
* `--requestheader-extra-headers-prefix=X-Remote-Extra-`
* `--requestheader-allowed-names=front-proxy-client`
### [controller manager](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#controller-manager)
#### [static Pod manifest](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/)
> \[!TIP]
>
> * If kubeadm is invoked specifying a `--pod-network-cidr`, the subnet manager feature required for some CNI network plugins is enabled by setting:
> * `--allocate-node-cidrs=true`
> * `--cluster-cidr` and `--node-cidr-mask-size` flags according to the given CIDR
> * If a cloud provider is specified, the corresponding `--cloud-provider` is specified, together with the `--cloud-config` path if such configuration file exists (this is experimental, alpha and will be removed in a future version)
#### other flags
* `--controllers` enabling all the default controllers plus `BootstrapSigner` and `TokenCleaner` controllers for TLS bootstrap. See TLS Bootstrapping for more details
* `--use-service-account-credentials` to true
* Flags for using certificates generated in previous steps:
* `--root-ca-file` to ca.crt
* `--cluster-signing-cert-file` to `ca.crt`, if External CA mode is disabled, otherwise to `""`
* `--cluster-signing-key-file` to `ca.key`, if External CA mode is disabled, otherwise to `""`
* `--service-account-private-key-file` to sa.key
### flow
#### pod creation
[](https://medium.com/jorgeacetozi/kubernetes-master-components-etcd-api-server-controller-manager-and-scheduler-3a0179fc8186)
#### ingress traffic
## [ports and protocols](https://kubernetes.io/docs/reference/ports-and-protocols/)
### contol plane
| PROTOCOL | DIRECTION | PORT RANGE | PURPOSE | USED BY |
| -------- | --------- | ----------- | ----------------------- | -------------------- |
| TCP | Inbound | `6443` | Kubernetes API server | All |
| TCP | Inbound | `2379-2380` | etcd server client API | kube-apiserver, etcd |
| TCP | Inbound | `10250` | Kubelet API | Self, Control plane |
| TCP | Inbound | `10259` | kube-scheduler | Self |
| TCP | Inbound | `10257` | kube-controller-manager | Self |
### worker node(s)
| PROTOCOL | DIRECTION | PORT RANGE | PURPOSE | USED BY |
| -------- | --------- | ------------- | ------------------------------------------------------------------------------------- | ------------------- |
| TCP | Inbound | `10250` | Kubelet API | Self, Control plane |
| TCP | Inbound | `30000-32767` | [NodePort Services](https://kubernetes.io/docs/concepts/services-networking/service/) | All |
## [architecture](https://devopscube.com/kubernetes-architecture-explained/)
### control pannel
#### kube-apiserver
#### etcd
#### kube-scheduler
#### controller manager
#### ccm : cloud controller manager
### work node
> \[!NOTE]
>
> * [supported endpoints](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
> * linux
| RUNTIME | PATH TO UNIX DOMAIN SOCKET |
| --------------------------------- | ------------------------------------------ |
| containerd | unix:///var/run/containerd/containerd.sock |
| CRI-O | unix:///var/run/crio/crio.sock |
| Docker Engine (using cri-dockerd) | unix:///var/run/cri-dockerd.sock |
> * windows
| RUNTIME | PATH TO UNIX DOMAIN SOCKET |
| --------------------------------- | -------------------------------------- |
| containerd | npipe:////./pipe/containerd-containerd |
| Docker Engine (using cri-dockerd) | npipe:////./pipe/cri-dockerd |
#### kubelet
#### kube proxy
#### cri-o : container runtime
## jsonpath
> \[!NOTE|label:references:]
>
> * [JSONPath Support](https://kubernetes.io/docs/reference/kubectl/jsonpath/)
> * [how to make kubectl jsonpath output on separate lines](https://downey.io/notes/dev/kubectl-jsonpath-new-lines/)
> * [JSONPath Reference](https://support.smartbear.com/readyapi/docs/testing/jsonpath-reference.html)
> * [Basic JSONPath Rules](https://docs.vmware.com/en/VMware-Carbon-Black-Cloud/services/carbon-black-cloud-user-guide/GUID-FBC6C73E-3807-4207-96FF-CF7EE7CE7AB8.html)
> * [Playing with kubectl output](https://gist.github.com/so0k/42313dbb3b547a0f51a547bb968696ba)
## options
### explain
```bash
$ kubectl explain hpa
KIND: HorizontalPodAutoscaler
VERSION: autoscaling/v1
DESCRIPTION:
configuration of a horizontal pod autoscaler.
FIELDS:
apiVersion
...
```
* or
```bash
$ kubectl explain configmap
KIND: ConfigMap
VERSION: v1
DESCRIPTION:
ConfigMap holds configuration data for pods to consume.
FIELDS:
apiVersion
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
...
```
## [kubectl alias](https://learnk8s.io/blog/kubectl-productivity/)
### `__start_kubectl`
```bash
$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ cat >> ~/.bashrc <> ~/.bashrc << EOF
command -v kubectl >/dev/null && source <(kubectl completion bash)
test -f ~/.bash_completion.sh && source ~/.bash_completion.sh
# or
# test -f /etc/profile.d/complete_alias.sh && source /etc/profile.d/complete_alias.sh
alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
alias km='kubectl -n monitoring'
complete -o default -F __start_kubectl kubecolor
complete -o nosort -o bashdefault -o default -F _complete_alias $(alias | sed -rn 's/^alias ([^=]+)=.+kubec.+$/\1/p' | xargs)
EOF
$ source ~/.bashrc
```
### kubecolor
```bash
$ [[ -d /tmp/kubecolor ]] && sudo mkdir -p /tmp/kubecolor
$ curl -fsSL https://github.com/hidetatz/kubecolor/releases/download/v0.0.25/kubecolor_0.0.25_Linux_x86_64.tar.gz | tar xzf - -C /tmp/kubecolor
$ sudo mv /tmp/kubecolor/kubecolor /usr/local/bin/
$ sudo chmod +x /usr/local/bin/kubecolor
```
## token
### check token
```bash
$ sudo kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
bop765.brol9nsrw820gmbi authentication,signing system:bootstrappers:kubeadm:default-node-token
khhfwa.jvkvrpiknx4o6ffy 19h 2018-07-13T11:37:43+08:00 authentication,signing system:bootstrappers:kubeadm:default-node-token
```
### generate token
> \[!NOET|label:see also:]
>
> * [\* iMarslo: get join command](https://imarslo.gitbook.io/book/virtualization/init/kubeadm#get-join-command)
> * [\* iMarslo: retrive join command](https://imarslo.gitbook.io/book/virtualization/init/kubeadm#retrive-join-command)
```bash
$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.1.100:6443 --token lhb1ln.oj0fqwgd1yl7l9xp --discovery-token-ca-cert-hash sha256:cba8df87dcb70c83c19af72c02e4886fcc7b0cf05319084751e6ece688443bde
$ sudo kubeadm token create --print-join-command --ttl=0
kubeadm join 192.168.1.100:6443 --token bop765.brol9nsrw820gmbi --discovery-token-ca-cert-hash sha256:c8650c56faf72b8bf71c576f0d13f44c93bea2d21d4329c64bb97cba439af5c3
```
## [tear down](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#tear-down)
> \[!TIP]
>
> * [How to completely uninstall kubernetes](https://stackoverflow.com/a/71503087/2940319)
> * [\* iMarslo: kubeadm reset and teardown](https://imarslo.gitbook.io/book/virtualization/init/kubeadm#kubeadm-reset-and-teardown)
* ubuntu
```bash
$ kubectl drain --delete-local-data --force --ignore-daemonsets
$ kubectl delete node
$ sudo kubeadm reset
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
$ systemctl stop kubelet
$ docker system prune -a -f
$ systemctl stop docker
$ sudo rm -rf /etc/kubernetes/
$ sudo rm -rf /var/lib/cni/
$ sudo rm -rf /var/lib/kubelet/*
$ sudo rm -rf /etc/cni/
$ sudo ifconfig cni0 down
$ sudo ifconfig flannel.1 down
$ rm -rf ~/.kube/
$ sudo apt purge kubeadm kubectl kubelet kubernetes-cni kube*
$ sudo apt autoremove
```
* CentOS/RHEL
```bash
$ kubectl drain --delete-local-data --force --ignore-daemonsets
$ kubectl delete node
$ sudo kubeadm reset -f --v=5
$ docker system prune -a -f
# stop and disable services
$ systemctl stop kubelet
$ systemctl disable kubelet
$ systemctl stop docker
$ systemctl disable docker
$ systemctl stop crio
# or
$ systemctl disable crio
$ sudo rm -rf /etc/systemd/system/multi-user.target.wants/kubelet.service
$ sudo rm -rf /etc/systemd/system/multi-user.target.wants/docker.service
$ sudo rm -rf /usr/lib/systemd/system/docker.service
$ sudo rm -rf /usr/lib/systemd/system/kubelet.service.d/
# network interface
$ sudo ifconfig cni0 down
$ sudo ip link delete cni0
$ sudo ifconfig flannel.1 down
$ sudo ip link delete flannel.1
$ sudo ifconfig docker0 down
$ sudo ip link delete docker0
$ sudo ifconfig vxlan.calico down
$ sudo ip link delete vxlan.calico
$ sudo yum versionlock delete docker-ce
$ sudo yum versionlock delete docker-ce-cli
$ sudo yum versionlock delete kubeadm
$ sudo yum versionlock delete kubelet
$ sudo yum versionlock delete kubectl
$ sudo yum versionlock delete kubernetes-cni
# or
$ sudo yum versionlock clear
$ sudo yum remove -y docker-ce docker-ce-cli containerd.io kubectl kubeadm kubelet kubernetes-cni
$ sudo yum autormeove
$ sudo rm -rf /etc/cni /etc/kubernetes /etc/docker $HOME/.kube
$ sudo rm -rf /usr/libexec/docker /usr/libexec/kubernetes
$ sudo rm -rf /var/lib/etcd/ # optional
$ sudo rm -rf /var/lib/kubelet/ /var/lib/dockershim /var/lib/yum/repos/x86_64/7/kubernetes /var/log/pods /var/log/containers
$ sudo rm -rf /var/run/docker.sock
$ sudo rm -rf /var/cache/yum/x86_64/7/kubernetes
$ sudo yum clean all
$ sudo rm -rf /var/cache/yum
$ sudo yum makecache
$ sudo yum check-update
```
## references
{% hint style="info" %}
> * references:
> * [\* Play with Kubernetes](https://labs.play-with-k8s.com/)
> * [\* Glossary](https://kubernetes.io/docs/reference/glossary/?fundamental=true)
> * [\* Install Kubernetes Cluster on Centos 8 With Kubeadm and CRI-O](https://tayeh.me/posts/install-kubernetes-cluster-on-centos-8-with-kubeadm-crio/)
> * [\* 在 CentOS 上部署 Kubernetes 集群](https://jimmysong.io/kubernetes-handbook/practice/install-kubernetes-on-centos.html)
> * [\* Kubernetes 中文指南/云原生应用架构实战手册](https://jimmysong.io/kubernetes-handbook/)
> * [\* Create static Pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/)
> * [\* Implementation details](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#core-design-principles)
> * [\* Scheduling, Preemption and Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/_print/)
> * [\* Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
> * [\* Administer a Cluster](https://kubernetes.io/docs/tasks/administer-cluster/)
> * [\* Kubernetes概念辨析](https://cloud-atlas.readthedocs.io/zh_CN/latest/kubernetes/concepts/index.html)
> * [\* Step by step installation of 3-nodes Kubernetes Cluster on CentOS 8](https://akyriako.medium.com/how-to-install-a-kubernetes-cluster-on-centos-8-1f9c5acd9c0e)
> * [\* KUBERNETES INSTALLATION ON CENTOS 8](https://www.topcoder.com/thrive/articles/kubernetes-installation-on-centos-8)
> * [\* Reference > Configuration APIs](https://kubernetes.io/docs/reference/config-api/)
> * [KUBERNETES: AN OVERVIEW](https://thenewstack.io/kubernetes-an-overview)
> * [What Is Kubernetes: A Container Orchestration Platform](https://www.metricfire.com/blog/what-is-kubernetes-a-container-orchestration-platform/)
> * [KUBERNETES, OPEN-SOURCE CONTAINER ORCHESTRATION TECHNOLOGY](https://quintagroup.com/cms/technology/kubernetes)
> * [Boosting your kubectl productivity](https://learnk8s.io/blog/kubectl-productivity/)
> * [23 Advanced kubectl commands](https://medium.com/faun/kubectl-commands-cheatsheet-43ce8f13adfb)
> * [8 Kubernetes Tips and Tricks](https://www.ibm.com/cloud/blog/8-kubernetes-tips-and-tricks)
> * [Linux namespace简介](https://blog.csdn.net/chenleiking/article/details/87915185)
> * [Well-Known Labels, Annotations and Taints](https://kubernetes.io/docs/reference/labels-annotations-taints)
> * [\* best practices](https://kubernetes.io/docs/setup/best-practices/_print/)
> * [12 Kubernetes Configuration Best Practices](https://cloud.redhat.com/blog/12-kubernetes-configuration-best-practices)
> * [Best Kubernetes Certifications for 2023 \[Ranked\]](https://devopscube.com/best-kubernetes-certifications/)
> * [Important Kubernetes Cluster Configurations](https://devopscube.com/kubernetes-cluster-configurations/)
> * Kubernetes Architecture
> * [Kubernetes Architecture Explained \[Comprehensive Guide\]](https://devopscube.com/kubernetes-architecture-explained/)
> * [\* How to Access Kubernetes API Server](https://blog.codefarm.me/2021/12/18/access-kubernetes/)
> * [Kubernetes（k8s）基础概念介绍](https://www.cnblogs.com/liugp/p/16345473.html)
> * [Migrating a cluster from flannel to Calico](https://docs.projectcalico.org/v3.9/getting-started/kubernetes/installation/migration-from-flannel)
> * [Kubernetes 网络组件](https://linux.cn/article-10003-1.html)
> * resources:
> * [Free Kubernetes Ebook: Kubernetes up and running](https://devopscube.com/free-kubernetes-ebook/)
> * [How to Learn Kubernetes (Complete Roadmap & Resources)](https://devopscube.com/learn-kubernetes-complete-roadmap/)
> * [kelseyhightower/kubernetes-the-hard-way](https://github.com/kelseyhightower/kubernetes-the-hard-way)
> * [\* Kubernetes Tutorials For Beginners: 43 Comprehensive Guides](https://devopscube.com/kubernetes-tutorials-beginners/)
> * [\* How to Setup Jenkins Build Agents on Kubernetes Pods](https://devopscube.com/jenkins-build-agents-kubernetes/)
> * [Kubernetes Learning Resources](https://devopscube.com/kubernetes/)
> * [Kubernetes Documentation/Tasks](https://kubernetes.io/docs/tasks/)
> {% endhint %}
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://imarslo.gitbook.io/book/virtualization/kubernetes.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.