troubleshooting

• error code

• debug services

  • get podIp

• check log

  • system logs

  • pod logs

[!TIP]

• * coredns 被误删了，可以通过重新应用 coredns 的 Deployment 或 DaemonSet 配置文件来恢复

• * k8s 中如何修改 pod-network-cidr 地址范围

• Troubleshooting Applications

  • Debug Pods

  • Debug Service

  • Debug a StatefulSet

  • Determine the Reason for Pod Failure

  • Debug Init Containers

  • Debug Running Pods

  • Get a Shell to a Running Container

• Troubleshooting Clusters

  • Resource metrics pipeline

  • Tools for Monitoring Resources

  • Monitor Node Health

  • Debugging Kubernetes nodes with crictl

  • Debugging Kubernetes Nodes With Kubectl

  • Developing and debugging services locally using telepresence

  • Auditing

  • Windows debugging tips

• Communicate Between Containers in the Same Pod Using a Shared Volume

• Translate a Docker Compose File to Kubernetes Resources

• pod资源限制和QoS探索

• 【K8S系列】Pod重启策略及重启可能原因

• POD的内存限制 和 OOM Killer

• 记一次k8s pod频繁重启的优化之旅

error code

ERROR CODE	COMMENTS
OOMKilled	pod运行过程内存需求持续增加超过为pod设置的内存大小
Pending	调度不成功 or 资源不足 or HostPort 已被占用
Waiting/ContainerCreating	镜像拉取失败 or CNI网络错误 or 容器无法启动 or 磁盘坏道input/output error
CrashLoopBackOff	容器曾经启动了但又异常退出
ImagePullBackOff	镜像名称配置错误或者私有镜像的密钥配置错误导致
CrashLoopBackOff	容器退出kubelet正在将它重启
InvalidImageName	无法解析镜像名称
ImageInspectError	无法校验镜像
ErrImageNeverPull	策略禁止拉取镜像
ImagePullBackOff	正在重试拉取
RegistryUnavailable	连接不到镜像中心
ErrImagePull	通用的拉取镜像出错
CreateContainerConfigError	不能创建kubelet使用的容器配置
CreateContainerError	创建容器失败
m.internalLifecycle.PreStartContainer	执行hook报错
RunContainerError	启动容器失败
PostStartHookError	执行hook报错
ContainersNotInitialized	容器没有初始化完毕
ContainersNotReady	容器没有准备完毕
ContainerCreating	容器创建中
PodInitializingpod	初始化中
DockerDaemonNotReady	docker还没有完全启动
NetworkPluginNotReady	网络插件还没有完全启动
Evicte	pod被驱赶

debug services

[!NOTE|label:reference:]

• How to Debug a Kubernetes Service Effectively
• Debug Services
• Access Services Running on Clusters
• DNS for Services and Pods
• svc in cluster can be visit via

  • CLUSTER-IP

  • <svc-name>.<namespace>.svc.cluster.local

- [create pod from cmd](./pod.html#pod) - svc status ```bash $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jenkins ClusterIP 10.111.230.13 8080/TCP,50017/TCP 17h ```

# create new pod
$ kubectl run ubuntu-marslo \
              --image=ubuntu:18.04 \
              --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname": "k8s-node-01"}}}' \
              -it \
              --rm

# check DNS
<ubuntu-marslo> $ cat /etc/resolv.conf
nameserver 10.96.0.10
search devops.svc.cluster.local svc.cluster.local cluster.local company.com
options ndots:5

# debug
$ nc -zv jenkins.devops.svc.cluster.local 30338
$ nc -zv 10.111.230.13                    30338
$ ssh -l marslo -p 30338 -i ~/.ssh/id_rsa jenkins.devops.svc.cluster.local list-plugins
$ ssh -l marslo -p 30338 -i ~/.ssh/id_rsa 10.111.230.13                    list-plugins

get podIp

$ kubectl get pods \
              -l app=hostnames \
              -o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'

check log

[!NOTE|label:references:]

• Kubernetes Logging Tutorial For Beginners

system logs

$ journalctl -u <service> -f

# or
$ journalctl -u kubelet -o cat

# or
$ sudo systemctl status <service> -l --no-pager

pod logs

references:

• Kubernetes / kubectl - "A container name must be specified" but seems like it is?

$ kubectl logs pod <pod_name> --all-containers