# crio

* [install](#install)
  * [prepare](#prepare)
  * [install package](#install-package)
  * [verify](#verify)
* [config](#config)
* [crictl](#crictl)
  * [remove](#remove)
* [tips](#tips)
  * [get cgroups](#get-cgroups)
* [troubleshooting](#troubleshooting)
  * [cni0](#cni0)

> \[!NOTE|label:references:]
>
> * [cri-o/cri-o](https://github.com/cri-o/cri-o)
> * [Using the CRI-O Container Engine](https://docs.openshift.com/container-platform/3.11/crio/crio_runtime.html)
> * [kubernetes/misc/kubernetes-with-crio](https://github.com/justmeandopensource/kubernetes/tree/master/misc/kubernetes-with-crio)
> * [\* Debugging Kubernetes nodes with crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/)
> * [在Kubernetes中使用CRI-O运行时](https://juejin.cn/post/6999405898980392996)

### install

#### prepare

*   enable kernel

    ```bash
    $ sudo modprobe overlay
    $ sudo modprobe br_netfilter
    ```
*   modify kernel parameters

    ```bash
    $ sudo bash -c "cat >>/etc/sysctl.d/99-kubernetes-crio.conf" << EOF
    net.ipv4.ip_forward = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    EOF
    $ sudo sysctl --system

    # or
    $ sudo sysctl -w net.ipv4.ip_forward=1
    $ sudo sysctl -w net.bridge.bridge-nf-call-iptables=1
    $ sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1
    ```

#### install package

> \[!NOTE|label:references:]
>
> * [CRI-O Packaging](https://github.com/cri-o/packaging/blob/main/README.md)
> * [cri-o/packaging actions](https://github.com/cri-o/packaging/actions)
> *   veriables:
>
>     ```bash
>     $ CRIO_VERSION='v1.30'
>     ```

*   rhel

    ```bash
    $ cat <<EOF | sudo tee /etc/yum.repos.d/cri-o.repo
    [cri-o]
    name=CRI-O
    baseurl=https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/rpm/
    enabled=1
    gpgcheck=1
    gpgkey=https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/rpm/repodata/repomd.xml.key
    exclude=cri-o
    EOF

    $ sudo bash -c "cat >>/etc/modules-load.d/crio.conf" << EOF
    overlay
    br_netfilter
    EOF

    $ sudo dnf install -y container-selinux
    $ sudo dnf install -y cri-o --disableexcludes=cri-o
    # or
    $ sudo dnf install -y cri-o-1.30.3-150500.1.1.x86_64 --disableexcludes=cri-o

    $ sudo systemctl enable --now crio.service

    # lock cri-o from auto upgrade
    $ sudo tail -1 /etc/yum.repos.d/cri-o.repo
    exclude=cri-o
    ```
*   debian

    ```bash
    $ sudo apt-get update
    $ sudo apt-get install -y software-properties-common curl

    $ curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key |
        gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
    $ echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" |
        tee /etc/apt/sources.list.d/cri-o.list

    $ apt-get update
    $ apt-get install -y cri-o
    ```
*   static binary bundles

    ```bash
    # latest
    $ curl https://raw.githubusercontent.com/cri-o/packaging/main/get | bash

    # specific version
    $ curl https://raw.githubusercontent.com/cri-o/packaging/main/get | bash -s -- -t v1.30.0

    # specific architectures
    $ curl https://raw.githubusercontent.com/cri-o/packaging/main/get | bash -s -- -a arm64
    ```

#### verify

> \[!NOTE|label:references:]
>
> * [#2416 - server status: add cgroup manager](https://github.com/cri-o/cri-o/pull/2416)

```bash
$ curl -v --unix-socket /var/run/crio/crio.sock http://localhost/info
*   Trying /var/run/crio/crio.sock...
* Connected to localhost (/var/run/crio/crio.sock) port 80 (#0)
> GET /info HTTP/1.1
> Host: localhost
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 06 Sep 2024 21:49:40 GMT
< Content-Length: 258
<
* Connection #0 to host localhost left intact
{"storage_driver":"overlay","storage_image":"","storage_root":"/var/lib/containers/storage","cgroup_driver":"systemd","default_id_mappings":{"uids":[{"container_id":0,"host_id":0,"size":4294967295}],"gids":[{"container_id":0,"host_id":0,"size":4294967295}]}}

# or
$ crio status info
INFO[2024-09-06 19:23:21.790806581-07:00] Starting CRI-O, version: 1.30.4, git: dbc00ffd41a487c847158032193b6dca9b49e821(clean)
cgroup driver: systemd
storage driver: overlay
storage graph root: /var/lib/containers/storage
storage image:
default GID mappings (format <container>:<host>:<size>):
  0:0:4294967295
default UID mappings (format <container>:<host>:<size>):
  0:0:4294967295
```

### config

*   `/etc/crictl.yaml`

    ```bash
    $ sudo cat /etc/crictl.yaml
    runtime-endpoint: "unix:///var/run/crio/crio.sock"
    timeout: 0
    debug: false
    ```

    *   [or](https://github.com/cri-o/cri-o/issues/3791#issuecomment-826867283)

        ```bash
        $ sudo cat /etc/crictl.yaml
        runtime-endpoint: "unix:///run/crio/crio.sock"
        image-endpoint: "unix:///run/crio/crio.sock"
        cgroup-manager: cgroupfs

        timeout: 10
        debug: true
        pull-image-on-create: false
        ```
*   [`/etc/modules-load.d/crio.conf`](https://github.com/justmeandopensource/kubernetes/tree/master/misc/kubernetes-with-crio)

    ```bash
    $ sudo cat /etc/modules-load.d/crio.conf
    overlay
    br_netfilter

    $ sudo modprobe overlay
    $ sudo modprobe br_netfilter
    ```
*   [`/etc/crio/crio.conf.d/02-cgroup-manager.conf`](https://github.com/justmeandopensource/kubernetes/tree/master/misc/kubernetes-with-crio)

    ```bash
    $ sudo cat /etc/crio/crio.conf.d/02-cgroup-manager.conf
    [crio.runtime]
    conmon_cgroup = "pod"
    cgroup_manager = "cgroupfs"

    # or using `systemd` as default cgroup manager
    $ sudo cat /etc/crio/crio.conf.d/02-cgroup-manager.conf
    [crio.runtime]
    conmon_cgroup = "pod"

    $ sudo systemctl daemon-reload
    $ sudo systemctl restart crio.service
    ```
*   [`/etc/cni/net.d/11-crio-ipv4-bridge.conflist`](https://blog.csdn.net/weixin_43323107/article/details/128535741)

    > \[!NOTE|label:references:]
    >
    > * [cri-o/cri-o](https://github.com/cri-o/cri-o/blob/main/contrib/cni/11-crio-ipv4-bridge.conflist)

    ```bash
    $ cat /etc/cni/net.d/11-crio-ipv4-bridge.conflist
    {
      "cniVersion": "1.0.0",
      "name": "crio",
      "plugins": [
        {
          "type": "bridge",
          "bridge": "cni0",
          "isGateway": true,
          "ipMasq": true,
          "hairpinMode": true,
          "ipam": {
            "type": "host-local",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ],
            "ranges": [
                [{ "subnet": "10.85.0.0/16" }]
            ]
          }
        }
      ]
    }

    # modify
    $ sudo sed -i 's/10.85.0.0/10.185.0.0/g' /etc/cni/net.d/11-crio-ipv4-bridge.conflist
    $ sudo systemctl daemon-reload
    $ sudo systemctl restart crio.service
    ```
*   [`/etc/crio/crio.conf.d/01-metrics.conf`](https://github.com/cri-o/cri-o/blob/main/tutorials/metrics.md)

    ```bash
    $ sudo cat /etc/crio/crio.conf.d/01-metrics.conf
    [crio.metrics]
    enable_metrics = true
    # default is 9090
    metrics_port = 9090
    ```

    *   verify:

        ```bash
        $ curl localhost:9090/metrics | grep -v '^#'
        ```

        ```bash
        $ curl -s localhost:9090/metrics | grep -v '^#'
        container_runtime_crio_containers_events_dropped_total 0
        container_runtime_crio_containers_oom_total 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="1000"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="1e+06"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="1e+07"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="5e+07"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="1e+08"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="2e+08"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="3e+08"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="4e+08"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="5e+08"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="1e+09"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="1e+10"} 0
        container_runtime_crio_image_pulls_layer_size_bucket{le="+Inf"} 0
        container_runtime_crio_image_pulls_layer_size_sum 0
        container_runtime_crio_image_pulls_layer_size_count 0
        container_runtime_crio_image_pulls_success_total 0
        container_runtime_crio_operations_errors_total{operation="ContainerStatus"} 5
        container_runtime_crio_operations_latency_seconds{operation="ContainerStatus"} 1.1523e-05
        container_runtime_crio_operations_latency_seconds{operation="CreateContainer"} 0.475484217
        container_runtime_crio_operations_latency_seconds{operation="ExecSync"} 0.090543277
        container_runtime_crio_operations_latency_seconds{operation="ImageFsInfo"} 0.00072969
        container_runtime_crio_operations_latency_seconds{operation="ImageStatus"} 0.000149831
        container_runtime_crio_operations_latency_seconds{operation="ListContainers"} 4.0299e-05
        container_runtime_crio_operations_latency_seconds{operation="ListImages"} 0.000504493
        container_runtime_crio_operations_latency_seconds{operation="ListPodSandbox"} 3.8221e-05
        container_runtime_crio_operations_latency_seconds{operation="PodSandboxStatus"} 2.5519e-05
        container_runtime_crio_operations_latency_seconds{operation="RemoveContainer"} 0.018241849
        container_runtime_crio_operations_latency_seconds{operation="RemoveImage"} 0.984503771
        container_runtime_crio_operations_latency_seconds{operation="RemovePodSandbox"} 0.001697702
        container_runtime_crio_operations_latency_seconds{operation="RunPodSandbox"} 0.123417422
        container_runtime_crio_operations_latency_seconds{operation="StartContainer"} 0.013815194
        container_runtime_crio_operations_latency_seconds{operation="Status"} 3.274e-05
        container_runtime_crio_operations_latency_seconds{operation="StopContainer"} 0.019264189
        container_runtime_crio_operations_latency_seconds{operation="StopPodSandbox"} 5.6215e-05
        container_runtime_crio_operations_latency_seconds{operation="Version"} 9.5257e-05
        container_runtime_crio_operations_latency_seconds{operation="network_setup_overall"} 0.117112005
        container_runtime_crio_operations_latency_seconds{operation="network_setup_pod"} 0.116541711
        container_runtime_crio_operations_latency_seconds_total{operation="ContainerStatus",quantile="0.5"} 1.7527e-05
        container_runtime_crio_operations_latency_seconds_total{operation="ContainerStatus",quantile="0.9"} 2.8625e-05
        container_runtime_crio_operations_latency_seconds_total{operation="ContainerStatus",quantile="0.99"} 4.0209e-05
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ContainerStatus"} 0.09174170199999988
        container_runtime_crio_operations_latency_seconds_total_count{operation="ContainerStatus"} 4672
        container_runtime_crio_operations_latency_seconds_total{operation="CreateContainer",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="CreateContainer",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="CreateContainer",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="CreateContainer"} 1.181032842
        container_runtime_crio_operations_latency_seconds_total_count{operation="CreateContainer"} 10
        container_runtime_crio_operations_latency_seconds_total{operation="ExecSync",quantile="0.5"} 0.085431263
        container_runtime_crio_operations_latency_seconds_total{operation="ExecSync",quantile="0.9"} 0.092410725
        container_runtime_crio_operations_latency_seconds_total{operation="ExecSync",quantile="0.99"} 0.154495402
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ExecSync"} 111.94222211099996
        container_runtime_crio_operations_latency_seconds_total_count{operation="ExecSync"} 813
        container_runtime_crio_operations_latency_seconds_total{operation="ImageFsInfo",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="ImageFsInfo",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="ImageFsInfo",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ImageFsInfo"} 0.001633457
        container_runtime_crio_operations_latency_seconds_total_count{operation="ImageFsInfo"} 2
        container_runtime_crio_operations_latency_seconds_total{operation="ImageStatus",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="ImageStatus",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="ImageStatus",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ImageStatus"} 0.005077591
        container_runtime_crio_operations_latency_seconds_total_count{operation="ImageStatus"} 21
        container_runtime_crio_operations_latency_seconds_total{operation="ListContainers",quantile="0.5"} 3.4691e-05
        container_runtime_crio_operations_latency_seconds_total{operation="ListContainers",quantile="0.9"} 4.667e-05
        container_runtime_crio_operations_latency_seconds_total{operation="ListContainers",quantile="0.99"} 6.4189e-05
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ListContainers"} 0.2532365459999996
        container_runtime_crio_operations_latency_seconds_total_count{operation="ListContainers"} 6666
        container_runtime_crio_operations_latency_seconds_total{operation="ListImages",quantile="0.5"} 0.000490407
        container_runtime_crio_operations_latency_seconds_total{operation="ListImages",quantile="0.9"} 0.000571354
        container_runtime_crio_operations_latency_seconds_total{operation="ListImages",quantile="0.99"} 0.000753937
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ListImages"} 0.3214459439999998
        container_runtime_crio_operations_latency_seconds_total_count{operation="ListImages"} 582
        container_runtime_crio_operations_latency_seconds_total{operation="ListPodSandbox",quantile="0.5"} 3.9654e-05
        container_runtime_crio_operations_latency_seconds_total{operation="ListPodSandbox",quantile="0.9"} 5.0074e-05
        container_runtime_crio_operations_latency_seconds_total{operation="ListPodSandbox",quantile="0.99"} 6.4293e-05
        container_runtime_crio_operations_latency_seconds_total_sum{operation="ListPodSandbox"} 0.25878698800000016
        container_runtime_crio_operations_latency_seconds_total_count{operation="ListPodSandbox"} 6193
        container_runtime_crio_operations_latency_seconds_total{operation="PodSandboxStatus",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="PodSandboxStatus",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="PodSandboxStatus",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="PodSandboxStatus"} 0.0011793539999999996
        container_runtime_crio_operations_latency_seconds_total_count{operation="PodSandboxStatus"} 37
        container_runtime_crio_operations_latency_seconds_total{operation="RemoveContainer",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RemoveContainer",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RemoveContainer",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="RemoveContainer"} 0.113517279
        container_runtime_crio_operations_latency_seconds_total_count{operation="RemoveContainer"} 9
        container_runtime_crio_operations_latency_seconds_total{operation="RemoveImage",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RemoveImage",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RemoveImage",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="RemoveImage"} 0.984530913
        container_runtime_crio_operations_latency_seconds_total_count{operation="RemoveImage"} 1
        container_runtime_crio_operations_latency_seconds_total{operation="RemovePodSandbox",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RemovePodSandbox",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RemovePodSandbox",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="RemovePodSandbox"} 0.013869093
        container_runtime_crio_operations_latency_seconds_total_count{operation="RemovePodSandbox"} 6
        container_runtime_crio_operations_latency_seconds_total{operation="RunPodSandbox",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RunPodSandbox",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="RunPodSandbox",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="RunPodSandbox"} 0.76309194
        container_runtime_crio_operations_latency_seconds_total_count{operation="RunPodSandbox"} 6
        container_runtime_crio_operations_latency_seconds_total{operation="StartContainer",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="StartContainer",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="StartContainer",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="StartContainer"} 0.18009194600000003
        container_runtime_crio_operations_latency_seconds_total_count{operation="StartContainer"} 10
        container_runtime_crio_operations_latency_seconds_total{operation="Status",quantile="0.5"} 2.3927e-05
        container_runtime_crio_operations_latency_seconds_total{operation="Status",quantile="0.9"} 3.4041e-05
        container_runtime_crio_operations_latency_seconds_total{operation="Status",quantile="0.99"} 4.2226e-05
        container_runtime_crio_operations_latency_seconds_total_sum{operation="Status"} 0.021624871000000014
        container_runtime_crio_operations_latency_seconds_total_count{operation="Status"} 809
        container_runtime_crio_operations_latency_seconds_total{operation="StopContainer",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="StopContainer",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="StopContainer",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="StopContainer"} 0.411804006
        container_runtime_crio_operations_latency_seconds_total_count{operation="StopContainer"} 6
        container_runtime_crio_operations_latency_seconds_total{operation="StopPodSandbox",quantile="0.5"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="StopPodSandbox",quantile="0.9"} NaN
        container_runtime_crio_operations_latency_seconds_total{operation="StopPodSandbox",quantile="0.99"} NaN
        container_runtime_crio_operations_latency_seconds_total_sum{operation="StopPodSandbox"} 1.156116755
        container_runtime_crio_operations_latency_seconds_total_count{operation="StopPodSandbox"} 12
        container_runtime_crio_operations_latency_seconds_total{operation="Version",quantile="0.5"} 8.5387e-05
        container_runtime_crio_operations_latency_seconds_total{operation="Version",quantile="0.9"} 0.000112424
        container_runtime_crio_operations_latency_seconds_total{operation="Version",quantile="0.99"} 0.000529564
        container_runtime_crio_operations_latency_seconds_total_sum{operation="Version"} 0.03793896699999996
        container_runtime_crio_operations_latency_seconds_total_count{operation="Version"} 396
        container_runtime_crio_operations_total{operation="ContainerStatus"} 4672
        container_runtime_crio_operations_total{operation="CreateContainer"} 10
        container_runtime_crio_operations_total{operation="ExecSync"} 813
        container_runtime_crio_operations_total{operation="ImageFsInfo"} 2
        container_runtime_crio_operations_total{operation="ImageStatus"} 21
        container_runtime_crio_operations_total{operation="ListContainers"} 6666
        container_runtime_crio_operations_total{operation="ListImages"} 582
        container_runtime_crio_operations_total{operation="ListPodSandbox"} 6193
        container_runtime_crio_operations_total{operation="PodSandboxStatus"} 37
        container_runtime_crio_operations_total{operation="RemoveContainer"} 9
        container_runtime_crio_operations_total{operation="RemoveImage"} 1
        container_runtime_crio_operations_total{operation="RemovePodSandbox"} 6
        container_runtime_crio_operations_total{operation="RunPodSandbox"} 6
        container_runtime_crio_operations_total{operation="StartContainer"} 10
        container_runtime_crio_operations_total{operation="Status"} 809
        container_runtime_crio_operations_total{operation="StopContainer"} 6
        container_runtime_crio_operations_total{operation="StopPodSandbox"} 12
        container_runtime_crio_operations_total{operation="Version"} 396
        container_runtime_crio_processes_defunct 0
        containerd_cri_input_bytes_total 0
        containerd_cri_input_entries_total 0
        containerd_cri_output_bytes_total 0
        containerd_cri_output_entries_total 0
        containerd_cri_split_entries_total 0
        go_gc_duration_seconds{quantile="0"} 5.5867e-05
        go_gc_duration_seconds{quantile="0.25"} 9.8548e-05
        go_gc_duration_seconds{quantile="0.5"} 0.00012298
        go_gc_duration_seconds{quantile="0.75"} 0.000165111
        go_gc_duration_seconds{quantile="1"} 0.000303457
        go_gc_duration_seconds_sum 0.012383793
        go_gc_duration_seconds_count 92
        go_goroutines 38
        go_info{version="go1.22.0"} 1
        go_memstats_alloc_bytes 1.1737336e+07
        go_memstats_alloc_bytes_total 6.76651384e+08
        go_memstats_buck_hash_sys_bytes 1.570182e+06
        go_memstats_frees_total 5.838505e+06
        go_memstats_gc_sys_bytes 3.905472e+06
        go_memstats_heap_alloc_bytes 1.1737336e+07
        go_memstats_heap_idle_bytes 9.37984e+06
        go_memstats_heap_inuse_bytes 1.7620992e+07
        go_memstats_heap_objects 86421
        go_memstats_heap_released_bytes 4.440064e+06
        go_memstats_heap_sys_bytes 2.7000832e+07
        go_memstats_last_gc_time_seconds 1.7260239921528594e+09
        go_memstats_lookups_total 0
        go_memstats_mallocs_total 5.924926e+06
        go_memstats_mcache_inuse_bytes 38400
        go_memstats_mcache_sys_bytes 46800
        go_memstats_mspan_inuse_bytes 447200
        go_memstats_mspan_sys_bytes 489600
        go_memstats_next_gc_bytes 1.8742896e+07
        go_memstats_other_sys_bytes 4.298866e+06
        go_memstats_stack_inuse_bytes 2.359296e+06
        go_memstats_stack_sys_bytes 2.359296e+06
        go_memstats_sys_bytes 3.9671048e+07
        go_threads 29
        process_cpu_seconds_total 11.68
        process_max_fds 1.048576e+06
        process_open_fds 58
        process_resident_memory_bytes 8.318976e+07
        process_start_time_seconds 1.72601996066e+09
        process_virtual_memory_bytes 3.351724032e+09
        process_virtual_memory_max_bytes 1.8446744073709552e+19
        promhttp_metric_handler_requests_in_flight 1
        promhttp_metric_handler_requests_total{code="200"} 6
        promhttp_metric_handler_requests_total{code="500"} 0
        promhttp_metric_handler_requests_total{code="503"} 0
        ```
*   `/etc/containers/registries.conf.d/01-unqualified.conf`

    ```bash
    unqualified-search-registries = ["docker.io", "quay.io"]
    ```

### crictl

#### remove

*   stop all pods

    ```bash
    $ crictl pods -q | xargs -r crictl -t 60s stopp

    # force remove all
    $ crictl rmp -a -f

    # or (rescue)
    $ ip netns list | cut -d' ' -f 1 | xargs -n1 ip netns delete && crictl rmp -a -f
    ```
*   remove all containers

    ```bash
    $ crictl ps -a -q | xargs crictl rm

    # or
    $ crictl rm -a -q
    ```

### tips

> \[!NOTE|label:references:]
>
> * [since CRI-O v1.18.0](https://github.com/cri-o/cri-o/releases/tag/v1.18.0)
> * [#3119 - remove NET\_RAW and SYS\_CHROOT capability by default](https://github.com/cri-o/cri-o/pull/3119)
>   * CRI-O now runs containers without `NET_RAW` and `SYS_CHROOT` capabilities by default.
>   * This can result in permission denied errors when the container tries to do something that would require either of these capabilities. For instance, using ping requires `NET_RAW`, unless the container is given the sysctl net.ipv4.ip\_forward.
> * [Kubernetes CRI-O Challenge | Ping permission denied | Are you root?](https://www.youtube.com/watch?v=ZKJ9oFwjosM)

#### get cgroups

```bash
$ curl -s --unix-socket /var/run/crio/crio.sock http://localhost/info | jq -r .cgroup_driver
systemd
```

### troubleshooting

#### cni0

> \[!NOTE|label:references:]
>
> *   [#6131 - contrib/cni/11-crio-ipv4-bridge.conf does not work out of the box](https://github.com/cri-o/cri-o/issues/6131)
>
>     ```bash
>     # debug via:
>     $ strace -s2048 -f -p $(pidof crio)
>     ```
> * [#2411 - cri-o w/ Kubernetes v1.14 connects pods to wrong subnets when using Weave or kube-router or Flannel CNI plugin](https://github.com/cri-o/cri-o/issues/2411)
> * [Kubernetes之network: failed to set bridge addr: “cni0“ already has an IP address different from xxx问题](https://blog.csdn.net/qq_40460909/article/details/114706367)
> * [解决k8s"failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24"](https://blog.csdn.net/Wuli_SmBug/article/details/104712653)
> * [#3555 - crio on minikube: could not add IP address to "cni0": permission denied](https://github.com/cri-o/cri-o/issues/3555)
> * [#5799 - what the Additional steps for crio when install Third party network plugin like calico flannel](https://github.com/cri-o/cri-o/issues/5799)
> * [#2572 - kubeadm init with pod-network-cidr but still remains using 10.85.0.0](https://github.com/kubernetes/kubeadm/issues/2572)
> * [#39557 - "Failed to setup network for pod \ using network plugins "cni": no IP addresses available in network: podnet; Skipping pod"](https://github.com/kubernetes/kubernetes/issues/39557)
