Summary
I have various single-node Kubernetes clusters which become unstable after having accumulated ~300 completed jobs.
In one cluster, for example, there are 303 completed jobs:
root@xxxx:/home/xxxx# kubectl get jobs | wc -l
303
Observations
What I observe is that
- The
kubelet
logs are filled with error messages like this:kubelet[877]: E0219 09:06:14.637045 877 reflector.go:134] object-"default"/"job-162273560": Failed to list *v1.ConfigMap: Get https://172.13.13.13:6443/api/v1/namespaces/default/configmaps?fieldSelector=metadata.name%3Djob-162273560&limit=500&resourceVersion=0: http2: no cached connection was available
- The node status is not being updated, with a similar error message:
kubelet[877]: E0219 09:32:57.379751 877 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://172.13.13.13:6443/api/v1/nodes?fieldSelector=metadata.name%3Dxxxxx&limit=500&resourceVersion=0: http2: no cached connection was available
- Eventually, the node is being marked as
NotReady
and no new pods are scheduledNAME STATUS ROLES AGE VERSION xxxxx NotReady master 6d4h v1.12.1
- The cluster is entering and exiting the master disruption mode (from the
kube-controller-manager
logs):I0219 09:29:46.875397 1 node_lifecycle_controller.go:1015] Controller detected that all Nodes are not-Ready. Entering master disruption mode. I0219 09:30:16.877715 1 node_lifecycle_controller.go:1042] Controller detected that some Nodes are Ready. Exiting master disruption mode.
The real culprit appears to be the http2: no cached connection was available
error message. The only real references I've found are a couple of issues in the Go repository (like #16582), which appear to have been fixed a long time ago.
In most cases, deleting the completed jobs seems to restore the system stability.
Minimal repro (tbc)
I seem to be able to reproduce this problem by creating lots of jobs which use containers which mount ConfigMaps:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: job-%JOB_ID%
data:
# Just some sample data
game.properties: |
enemies=aliens
lives=3
enemies.cheat=true
enemies.cheat.level=noGoodRotten
secret.code.passphrase=UUDDLRLRBABAS
secret.code.allowed=true
secret.code.lives=30
ui.properties: |
color.good=purple
color.bad=yellow
allow.textmode=true
how.nice.to.look=fairlyNice
---
apiVersion: batch/v1
kind: Job
metadata:
name: job-%JOB_ID%
spec:
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(20)"]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: job-%JOB_ID%
restartPolicy: Never
backoffLimit: 4
Schedule lots of these jobs:
#!/bin/bash
for i in `seq 100 399`;
do
cat job.yaml | sed "s/%JOB_ID%/$i/g" | kubectl create -f -
sleep 0.1
done
Questions
I'm very curious though as to what causes this problem, as 300 completed jobs seems to be a fairly low number.
Is this a configuration problem in my cluster? A possible bug in Kubernetes/Go? Anything else that I can try?