# addons

* [helm](#helm)
* [CNI](#cni)
  * [calico](#calico)
  * [flannel](#flannel)
* [ingress](#ingress)
  * [ingress-nginx](#ingress-nginx)
* [monitoring](#monitoring)
  * [metrics-server](#metrics-server)
  * [kubernetes-dashboard](#kubernetes-dashboard)
    * [ingress for kubernetes-dashboard](#ingress-for-kubernetes-dashboard)
    * [RBAC](#rbac)
  * [prometheus stack](#prometheus-stack)
    * [grafana](#grafana)
* [tls](#tls)
  * [import tls](#import-tls)
  * [copy tls](#copy-tls)

> \[!TIP|label:references]
>
> * [see also: iMarlso: tools](/book/virtualization/kubernetes/tools.md)

## helm

```bash
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Downloading https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
```

## CNI

### calico

```bash
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/tigera-operator.yaml
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/custom-resources.yaml

# verify
$ kubectl get pods -n calico-system
```

* calico tools

  > \[!NOTE|label:references:]
  >
  > * [install calicoctl](https://docs.tigera.io/calico/latest/operations/calicoctl/install)

  ```bash
  # calicoctl
  $ curl -L https://github.com/projectcalico/calico/releases/download/v3.28.1/calicoctl-linux-amd64 -o calicoctl
  $ chmod +x calicoctl
  $ sudo mv calicoctl /usr/local/bin/

  # kubectl-calico
  $ curl -L https://github.com/projectcalico/calico/releases/download/v3.28.1/calicoctl-linux-amd64 -o kubectl-calico
  $ chmod +x kubectl-calico
  $ sudo mv kubectl-calico /usr/local/bin/
  ```

### flannel

> \[!NOTE|label:references:]
>
> * [#2695 dial tcp 10.96.0.1:443 timeout](https://github.com/projectcalico/calico/issues/2695)
> * [k8s安装插件出现dial tcp 10.96.0.1:443: i/o timeout问题解析](https://blog.csdn.net/qq_44847649/article/details/123504663)
> * [kube-flannel.yml](https://www.cnblogs.com/chaojiyingxiong/p/16896593.html)
> * [Kubernetes / Flannel – Failed to list \*v1.Service](https://www.thenoccave.com/2020/08/kubernetes-flannel-failed-to-list-v1-service/)
>   * `$ cat /run/flannel/subnet.env`
>   * `$ kubectl get nodes k8s-node-01 -o jsonpath='{.spec.podCIDR}'`

* modify

  ```bash
  $ kubectl edit cm -n kube-system kube-flannel-cfg
  net-conf.json: |
       {
         "Network": "10.244.0.0/16",
         "Backend": {
           "Type": "vxlan"
         }
       }
  ```
* check

  ```bash
  $ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

  # e.g.: flannel
  $ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
  10.244.21.0/24 10.244.4.0/24 10.244.1.0/24 10.244.10.0/24 10.244.20.0/24 10.244.7.0/24 10.244.5.0/24 10.244.17.0/24 10.244.3.0/24 10.244.0.0/24 10.244.6.0/24 10.244.12.0/24 10.244.13.0/24 10.244.16.0/24 10.244.15.0/24
  ```

## ingress

> \[!NOTE|label:references:]
>
> * [入门级实操教程！从概念到部署，全方位了解 K8S Ingress](https://www.infoq.cn/article/vl7SZGQc2JUaM6IisAhE)
> * [k8s指南-Ingress](https://blog.csdn.net/xiaoyi52/article/details/128289679)

### ingress-nginx

```bash
$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
$ helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace

# or
$ helm upgrade --install ingress-nginx ingress-nginx \
       --repo https://kubernetes.github.io/ingress-nginx \
       --namespace ingress-nginx --create-namespace

# check value
$ helm show values ingress-nginx --repo https://kubernetes.github.io/ingress-nginx
```

## monitoring

### metrics-server

```bash
$ helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
$ helm upgrade --install metrics-server metrics-server/metrics-server --namespace monitoring --create-namespace

# without tls: https://github.com/kubernetes-sigs/metrics-server/issues/1221
$ helm upgrade metrics-server metrics-server/metrics-server --set args="{--kubelet-insecure-tls}" --namespace monitoring
```

### kubernetes-dashboard

> \[!NOTE|label:references:]
>
> * [Configure Kubernetes Dashboard Web UI hosted with Nginx Ingress Controller](https://gist.github.com/s-lyn/3aba97628c922ddc4a9796ac31a6df2d)
> * [Enabling Kubernetes Dashboard over HTTPS with RBAC Authorization](https://blog.zachinachshon.com/k8s-dashboard/)
>   * Admin User
>   * Cluster Admin User
>   * Ready-only User
>   * Access via kubeconfig

```bash
# add kubernetes-dashboard repository
$ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/

# deploy a Helm Release named "kubernetes-dashboard" using the kubernetes-dashboard chart
$ helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard
```

#### ingress for kubernetes-dashboard

```yaml
# v7.x
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/secure-backends: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - sms-k8s-dashboard.sample.com
    secretName: sample-tls
  rules:
    - host: sms-k8s-dashboard.sample.com
      http:
        paths:
        - path: /
          backend:
            service:
              name: kubernetes-dashboard-kong-proxy
              port:
                number: 443
          pathType: Prefix
```

```yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/secure-backends: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - sms-k8s-dashboard.sample.com
    secretName: sample-tls
  rules:
    - host: sms-k8s-dashboard.sample.com
      http:
        paths:
        - path: /
          backend:
            service:
              # or kubernetes-dashboard-kong-proxy for latest version
              name: kubernetes-dashboard
              port:
                number: 443
          pathType: Prefix
```

#### RBAC

[**create admin user for kubernetes-dashboard**](https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md)

> \[!NOTE|label:references:]
>
> * [Creating sample user](https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md#creating-sample-user)
>   * using `ClusterRole` : `cluster-admin` for `kubernetes-dashboard-admin`
> * [Kubernetes - Dashboard 配置用户名密码方式登录](https://blog.csdn.net/lookboydfw/article/details/138320111)

* create ServiceAccount in namespace

  ```bash
  $ kubectl -n monitoring create serviceaccount kubernetes-dashboard-admin
  ```
* create ClusterRoleBinding for ServiceAccount

  ```bash
  $ kubectl create clusterrolebinding kubernetes-dashboard-admin \
            --clusterrole=cluster-admin \                              # default ClusterRole
            --serviceaccount=monitoring:kubernetes-dashboard-admin     # service account

  $ kubectl get clusterrolebinding kubernetes-dashboard-admin -o yaml | grep -v -E 'uid:|resourceVersion:|creationTimestamp:'
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: kubernetes-dashboard-admin
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
  subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard-admin
    namespace: monitoring
  ```
* generate token

  ```bash
  # admin
  $ kubectl -n monitoring create token kubernetes-dashboard-admin

  # normal user
  $ kubectl -n monitoring create token kubernetes-dashboard-metrics-scraper
  ```
* [manually create a long-lived api token for a serviceaccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-a-long-lived-api-token-for-a-serviceaccount)

  ```bash
  $ kubectl -n monitoring apply -f - <<EOF
  apiVersion: v1
  kind: Secret
  metadata:
    name: kubernetes-dashboard-admin-token
    namespace: monitoring
    annotations:
      kubernetes.io/service-account.name: kubernetes-dashboard-admin
  type: kubernetes.io/service-account-token
  EOF

  # get token by service account
  $ kubectl -n monitoring get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='kubernetes-dashboard-admin')].data.token}" | base64 -d
  # or
  $ kubectl -n monitoring get secrets -o jsonpath="{.items[?(@.metadata.annotations.kubernetes\.io/service-account\.name=='kubernetes-dashboard-admin')].data.token}" | base64 -d

  # or get token by describe token
  $ kubectl -n monitoring describe secrets $(kubectl -n monitoring get secret | grep kubernetes-dashboard-admin | awk '{print $1}') | grep 'token' | awk '{print $2}'
  ```
* or modify ClusterRole `kubernetes-dashboard-metrics-scraper` manually

  > \[!TIP] for v7.x

  * clusterrole: `kubernetes-dashboard-metrics-scraper`

    ```yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      annotations:
        meta.helm.sh/release-name: kubernetes-dashboard
        meta.helm.sh/release-namespace: monitoring
      labels:
        app.kubernetes.io/instance: kubernetes-dashboard
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: kubernetes-dashboard
        helm.sh/chart: kubernetes-dashboard-7.5.0
      name: kubernetes-dashboard-metrics-scraper
    rules:
    - apiGroups:
      - '*'
      resources:
      - '*'
      verbs:
      - '*'
    ```

    * original

      ```yaml
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        annotations:
          meta.helm.sh/release-name: kubernetes-dashboard
          meta.helm.sh/release-namespace: monitoring
        labels:
          app.kubernetes.io/instance: kubernetes-dashboard
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/part-of: kubernetes-dashboard
          helm.sh/chart: kubernetes-dashboard-7.5.0
        name: kubernetes-dashboard-metrics-scraper
      rules:
      - apiGroups:
        - metrics.k8s.io
        resources:
        - pods
        - nodes
        verbs:
        - get
        - list
        - watch
      ```
  * clusterrolebinding: `kubernetes-dashboard-metrics-scraper`

    ```bash
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      annotations:
        meta.helm.sh/release-name: kubernetes-dashboard
        meta.helm.sh/release-namespace: monitoring
      labels:
        app.kubernetes.io/instance: kubernetes-dashboard
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: kubernetes-dashboard
        helm.sh/chart: kubernetes-dashboard-7.5.0
      name: kubernetes-dashboard-metrics-scraper
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: kubernetes-dashboard-metrics-scraper
    subjects:
    - kind: ServiceAccount
      name: kubernetes-dashboard-metrics-scraper
      namespace: monitoring
    ```
  * serviceaccount: `kubernetes-dashboard-metrics-scraper`

    ```bash
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      annotations:
        meta.helm.sh/release-name: kubernetes-dashboard
        meta.helm.sh/release-namespace: monitoring
      labels:
        app.kubernetes.io/instance: kubernetes-dashboard
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: kubernetes-dashboard
        helm.sh/chart: kubernetes-dashboard-7.5.0
      name: kubernetes-dashboard-metrics-scraper
      namespace: monitoring
    ```
  * generate token

    ```bash
    $ kubectl -n monitoring create token kubernetes-dashboard-metrics-scraper
    ey**********************WAA
    ```
  * older version
    * clusterrole

      ```bash
      $ kubectl get clusterrole kubernetes-dashboard -o yaml
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        labels:
          k8s-app: kubernetes-dashboard
        name: kubernetes-dashboard
      rules:
      - apiGroups:
        - '*'
        resources:
        - '*'
        verbs:
        - '*'
      ```
    * clusterrolebinding

      ```bash
      $ kubectl -n kube-system get clusterrolebindings kubernetes-dashboard -o yaml
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: kubernetes-dashboard
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: kubernetes-dashboard
      subjects:
      - kind: ServiceAccount
        name: kubernetes-dashboard
        namespace: kube-system
      ```
    * serviceaccount

      ```bash
      $ kubectl -n kube-system get sa kubernetes-dashboard -o yaml
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        labels:
          k8s-app: kubernetes-dashboard
        name: kubernetes-dashboard
        namespace: kube-system
      ```
    * generate token

      ```bash
      $ kubectl -n kube-system create token kubernetes-dashboard
      ey**********************WAA
      ```

### prometheus stack

> \[!NOTE|label:references:]
>
> * [assign different port for node-exporter in kube-prometheus-stack](https://stackoverflow.com/a/75427221/2940319)
> * [Prometheus+Grafana监控K8S集群(基于K8S环境部署)](https://blog.csdn.net/weixin_45623111/article/details/135381921)
> * [Grafana dashboard best practices](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/best-practices/):
>   * [\* iMarslo: grafana dashboard](/book/virtualization/kubernetes/tools/monitor.md#dashboard)
> * cluster level: 13332、13824、14518

```bash
$ helm repo add prometheus-stack https://prometheus-community.github.io/helm-charts
"prometheus-stack" has been added to your repositories

$ helm upgrade --install prometheus-stack prometheus-stack/kube-prometheus-stack --namespace monitoring

# if node-exporter has been deployed by kubespary, assign another port for kube-prometheus-stack
$ helm upgrade --install prometheus-stack prometheus-stack/kube-prometheus-stack \
               --namespace monitoring \
               --set prometheus-node-exporter.service.port=9200
Release "prometheus-stack" has been upgraded. Happy Helming!
NAME: prometheus-stack
LAST DEPLOYED: Tue Sep 10 22:53:40 2024
NAMESPACE: monitoring
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
```

**ingress**

* grafana

  > \[!NOTE|label:references]
  >
  > * [#1748 - Expose Grafana with Ingress](https://github.com/prometheus-operator/kube-prometheus/issues/1748)

  ```yaml
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: prometheus-stack-grafana
    namespace: monitoring
    annotations:
      kubernetes.io/ingress.class: "nginx"
      nginx.ingress.kubernetes.io/secure-backends: "true"
  spec:
    ingressClassName: nginx
    tls:
    - hosts:
      - sms-k8s-grafana.sample.com
      secretName: sample-tls
    rules:
      - host: sms-k8s-grafana.sample.com
        http:
          paths:
          - path: /
            backend:
              service:
                name: prometheus-stack-grafana
                port:
                  number: 80
            pathType: Prefix
  ```
* prometheus

  ```yaml
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: prometheus
    namespace: monitoring
    annotations:
      kubernetes.io/ingress.class: "nginx"
  spec:
    ingressClassName: nginx
    tls:
    - hosts:
      - sms-k8s-prometheus.marvell.com
      secretName: marvell-tls
    rules:
      - host: sms-k8s-prometheus.marvell.com
        http:
          paths:
          - path: /
            backend:
              service:
                name: prometheus-stack-kube-prom-prometheus
                port:
                  number: 9090
            pathType: Prefix
  ```
* alertmanager

  ```yaml
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: alertmanager
    namespace: monitoring
    annotations:
      kubernetes.io/ingress.class: "nginx"
  spec:
    ingressClassName: nginx
    tls:
    - hosts:
      - sms-k8s-alertmgr.marvell.com
      secretName: marvell-tls
    rules:
      - host: sms-k8s-alertmgr.marvell.com
        http:
          paths:
          - path: /
            backend:
              service:
                name: prometheus-stack-kube-prom-alertmanager
                port:
                  number: 9093
            pathType: Prefix
  ```

**admin account for grafana**

```bash
# account
$ kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsh='{.data.admin-user}' | base64 -d; echo

# password
$ kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
```

#### grafana

```bash
$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm repo list
NAME                    URL
kubernetes-dashboard    https://kubernetes.github.io/dashboard/
ingress-nginx           https://kubernetes.github.io/ingress-nginx
grafana                 https://grafana.github.io/helm-charts

$ helm repo update
$ helm search repo grafana/grafana

$ helm install grafana grafana/grafana --namespace monitoring --create-namespace
```

## tls

### import tls

```bash
$ sudo kubectl -n kube-system create secret tls sample-tls --cert star_sample_com.full.crt --key star_sample_com.key --dry-run=client -o yaml > kube-system.sample-tls.yaml
```

### copy tls

```bash
$ kubectl --namespace=kube-system get secrets sample-tls -o yaml | grep -v '^\s*namespace:\s' | kubectl apply --namespace=monitoring -f -
secret/sample-tls created

$ kubectl --namespace=kube-system get secrets sample-tls -o yaml | grep -v '^\s*namespace:\s' | kubectl apply --namespace=kubernetes-dashboard -f -
secret/sample-tls created

$ kubectl --namespace=kube-system get secrets sample-tls -o yaml | grep -v '^\s*namespace:\s' | kubectl apply --namespace=ingress-nginx -f -
secret/sample-tls created

$ kubectl --namespace=kube-system get secrets sample-tls -o yaml | grep -v '^\s*namespace:\s' | kubectl apply --namespace=default -f -
secret/sample-tls created
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://imarslo.gitbook.io/book/virtualization/kubernetes/init/addons.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.