When I trying to follow the Kubernetes Configure Multiple Schedulers guide, I cannot successfully get the my-scheduler pod running.
I succeed to build the image and push it to a registry, and the image shows in my docker and Google Could Platform.
But when I apply the deployment file to set the pods, the status is CrashLoopBackOff.
my-scheduler-64576b7897-8b549 0/1 CrashLoopBackOff 14 84m
Here is the output for kuberctl describe pod my-scheduler-64576b7897-8b549 -n kube-system:
Name: my-scheduler-64576b7897-8b549
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: minikube/10.0.2.15
Start Time: Tue, 07 May 2019 15:06:59 -0500
Labels: component=scheduler
pod-template-hash=64576b7897
tier=control-plane
version=second
Annotations: <none>
Status: Running
IP: 172.17.0.15
Controlled By: ReplicaSet/my-scheduler-64576b7897
Containers:
kube-second-scheduler:
Container ID: docker://a0cadf03dd5ffd11cbb6f1f8e669fc04cd6aa00aa6f361550f06f32785a4d63c
Image: gcr.io/my-gcp-project-239900/my-kube-scheduler:1.0
Image ID: docker-pullable://gcr.io/my-gcp-project-239900/my-kube-scheduler@sha256:0895687b7646d159438f086e8f8b23b253332930e169cf29d8b9951ebf3ba653
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/kube-scheduler
--address=1.1.1.1
--leader-elect=false
--scheduler-name=my-scheduler
State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 07 May 2019 15:07:45 -0500
Finished: Tue, 07 May 2019 15:07:45 -0500
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 07 May 2019 15:07:24 -0500
Finished: Tue, 07 May 2019 15:07:24 -0500
Ready: False
Restart Count: 3
Requests:
cpu: 100m
Liveness: http-get http://:10251/healthz delay=15s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:10251/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from my-scheduler-token-lzwvk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
my-scheduler-token-lzwvk:
Type: Secret (a volume populated by a Secret)
SecretName: my-scheduler-token-lzwvk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 48s default-scheduler Successfully assigned kube-system/my-scheduler-64576b7897-8b549 to minikube
Normal Pulled 2s (x4 over 47s) kubelet, minikube Container image "gcr.io/my-gcp-project-239900/my-kube-scheduler:1.0" already present on machine
Normal Created 2s (x4 over 47s) kubelet, minikube Created container kube-second-scheduler
Normal Started 2s (x4 over 47s) kubelet, minikube Started container kube-second-scheduler
Warning BackOff 1s (x6 over 45s) kubelet, minikube Back-off restarting failed container
And when I try to see the log of this pod, this error message appears:
standard_init_linux.go:207: exec user process caused "exec format error"
And my Dockerfile is bit different from the guide, it looks like:
FROM busybox
ADD ./_output/local/bin/darwin/amd64/kube-scheduler /usr/local/bin/kube-scheduler
I changed
ADD ./_output/dockerized/bin/linux/amd64/kube-scheduler /usr/local/bin/kube-scheduler
into
ADD ./_output/local/bin/darwin/amd64/kube-scheduler /usr/local/bin/kube-scheduler
because ./_output/local/bin/darwin/amd64/kube-scheduleris where the scheduler file acctually located. I would also appreciate it if anyone could tell me why would this happen.
Some Updates
Different file location
Following the guide, I typed these command in my shell
git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
make
by executing make, the Makefilefile will generate the kubernetes exec files in certain path (including the kube-scheduler we want). And the generation path is related to your system. Since I'm using minikube to run kubernetes and docker, and it runs on darwin(amd64). I think that's the reason why my path is different from the guide.
The "exec format error"
For this error
standard_init_linux.go:207: exec user process caused "exec format error"
I find this issue suggesting add the #!/bin/bash before you try to run commands in the container. This tells the system to treat the commands as bash commands. Here in busybox it should be /bin/sh. (I don't know much about shell language mechanism, so correct me if I'm wrong)
So I changed these lines in my-sheduler.yaml:
- command:
- /usr/local/bin/kube-scheduler
- --address=0.0.0.0
- --leader-elect=false
- --scheduler-name=my-scheduler
into
- command:
- /bin/sh
- /usr/local/bin/kube-scheduler
- --address=0.0.0.0
- --leader-elect=false
- --scheduler-name=my-schedule
Now the my-sheduler pods still fail and get the CrashLoopBackOff error.
Good news is I now have the log when executing kubectl logs.
Bad news is the logs are unreadable like this
/usr/local/bin/kube-scheduler: line 1: ����: not found
/usr/local/bin/kube-scheduler: line 2: syntax error: unexpected ")"
I assume this happened because of something happened in the docker image. But since I'm using the scrips right from the kubernetes github page, there shouldn't be any bug.
The same log shows up when I run docker run -it {my-image-id} /bin/sh and go into my image and run the exec kube-scheduler file.
I can't figure it out what's in the GO script causes this problem.
Another alternative
I ended up trying another way to build my custom scheduler.
I followed this tutorial: https://www.youtube.com/watch?v=IYcL0Un1io0&t=1807s
And you can find the source code here: https://github.com/kelseyhightower/scheduler
Comparing to the official guide, this one is more clear light (in in official guide you have to build the whole thing to get the scheduler file).
Hope this helps if someone is running into the same problem.
