问题描述： mysql-3173324660 创建后一直处于 ContainerCreating状态
[root@user1-group1-295 ceph]# kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-3173324660-jd8jw 0/1 ContainerCreating 0 1m
mysql-sonar-1457409014-hsbwg 1/1 Running 4 117d
kubectl describe po/mysql-3173324660-jd8jw
报挂卷失败：
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
2m 2m 1 default-scheduler Normal Scheduled Successfully assigned mysql-3173324660-jd8jw to user1-group1-296
34s 34s 1 kubelet, user1-group1-296 Warning FailedMount Unable to mount volumes for pod "mysql-3173324660-jd8jw_default(cc915019-9139-11e7-b200-fa163e9b5d77)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-3173324660-jd8jw". list of unattached/unmounted volumes=[mysql-persistent-storage]
34s 34s 1 kubelet, user1-group1-296 Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-3173324660-jd8jw". list of unattached/unmounted volumes=[mysql-persistent-storage]
2m 24s 9 kubelet, user1-group1-296 Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/cc915019-9139-11e7-b200-fa163e9b5d77-mysql-pv" (spec.Name: "mysql-pv") pod "cc915019-9139-11e7-b200-fa163e9b5d77" (UID: "cc915019-9139-11e7-b200-fa163e9b5d77") with: rbd: failed to mount rbd volume /dev/rbd4 [xfs] to /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-imagemysql1, error executable file not found in $PATH
按照楼下朋友指点需要查看的几个点,rdb正常，pv和pvc都是bound：
[root@user1-group1-295 ceph]# rbd list
caasmysqlimage
ceph-image
ceph-image-new
foo
image-mysql-wb1
imagemysql1
imagewb
imagewb1
imagewbb1
imagewbb2
jdk-image
mysql-image
mysql-image-1
[root@user1-group1-295 ceph]# kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
mysql-pv 2Gi RWO Recycle Bound default/mysql-pv-claim 12m
[root@user1-group1-295 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
mysql-pv-claim Bound mysql-pv 2Gi RWO 12m
[root@user1-group1-295 ceph]#
Sep 04 06:25:03 左右创建的pod：
创建pod失败时候的日志如下：
[root@user1-group1-295 ceph]# journalctl -u kubelet -f
-- Logs begin at Tue 2017-07-11 06:42:53 UTC. --
Sep 04 06:23:21 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:21.226456 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77").
Sep 04 06:23:22 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:22.229211 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77").
Sep 04 06:23:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:23.736718 16005 server.go:778] GET /metrics: (107.402434ms) 200 [[Go-http-client/1.1] 172.17.0.162:48938]
Sep 04 06:23:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:33.740537 16005 server.go:778] GET /metrics: (111.112116ms) 200 [[Go-http-client/1.1] 172.17.0.162:48960]
Sep 04 06:23:34 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:34.171001 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77").
Sep 04 06:23:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:43.736004 16005 server.go:778] GET /metrics: (106.587002ms) 200 [[Go-http-client/1.1] 172.17.0.162:48982]
Sep 04 06:23:44 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:44.201917 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77").
Sep 04 06:23:50 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:50.219064 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388").
Sep 04 06:23:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:53.227001 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77").
Sep 04 06:23:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:53.743604 16005 server.go:778] GET /metrics: (114.703271ms) 200 [[Go-http-client/1.1] 172.17.0.162:49004]
Sep 04 06:24:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:03.732337 16005 server.go:778] GET /metrics: (103.103667ms) 200 [[Go-http-client/1.1] 172.17.0.162:49026]
Sep 04 06:24:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:24:05.500408 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"}
Sep 04 06:24:06 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:06.262225 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fccdbfd7-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fccdbfd7-360d-11e7-a88c-fa163ed46388" (UID: "fccdbfd7-360d-11e7-a88c-fa163ed46388").
Sep 04 06:24:11 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:11.175838 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/05c2105e-4b6a-11e7-ad4d-fa163ed46388-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "05c2105e-4b6a-11e7-ad4d-fa163ed46388" (UID: "05c2105e-4b6a-11e7-ad4d-fa163ed46388").
Sep 04 06:24:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:13.740341 16005 server.go:778] GET /metrics: (110.717336ms) 200 [[Go-http-client/1.1] 172.17.0.162:49048]
Sep 04 06:24:15 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:15.989430 16005 container_manager_linux.go:439] Discovered runtime cgroups name: /system.slice/docker.service
Sep 04 06:24:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:23.740110 16005 server.go:778] GET /metrics: (110.727464ms) 200 [[Go-http-client/1.1] 172.17.0.162:49070]
Sep 04 06:24:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:33.736772 16005 server.go:778] GET /metrics: (107.362082ms) 200 [[Go-http-client/1.1] 172.17.0.162:49092]
Sep 04 06:24:34 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:34.247795 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f1b4decb-33af-11e7-a587-fa163ed46388-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "f1b4decb-33af-11e7-a587-fa163ed46388" (UID: "f1b4decb-33af-11e7-a587-fa163ed46388").
Sep 04 06:24:42 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:42.169852 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77").
Sep 04 06:24:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:43.739418 16005 server.go:778] GET /metrics: (110.330416ms) 200 [[Go-http-client/1.1] 172.17.0.162:49114]
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: E0904 06:24:48.725052 16005 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc with output stdout: 2730096 /var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: , stderr: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/task/30247/fd/4': No such file or directory
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/task/30247/fdinfo/4': No such file or directory
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/fd/4': No such file or directory
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/fdinfo/4': No such file or directory
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: - exit status 1, rootInodeErr: , extraDiskErr:
Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:48.725072 16005 fsHandler.go:131] du and find on following dirs took 1.057182281s: [/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc /var/lib/docker/containers/7d9aa1bfde24865c84c5a7e8577965741ec339379213af49f400deef4592803a]
Sep 04 06:24:50 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:50.194610 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77").
Sep 04 06:24:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:53.741418 16005 server.go:778] GET /metrics: (112.212086ms) 200 [[Go-http-client/1.1] 172.17.0.162:49136]
Sep 04 06:24:59 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:59.218706 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77").
Sep 04 06:25:02 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:02.226260 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77").
Sep 04 06:25:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:03.744718 16005 server.go:778] GET /metrics: (115.63146ms) 200 [[Go-http-client/1.1] 172.17.0.162:49158]
Sep 04 06:25:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:25:05.623008 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"}
Sep 04 06:25:06 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:06.243201 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77").
Sep 04 06:25:12 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:12.260590 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388").
Sep 04 06:25:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:13.737764 16005 server.go:778] GET /metrics: (108.38973ms) 200 [[Go-http-client/1.1] 172.17.0.162:49180]
Sep 04 06:25:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:23.727904 16005 server.go:778] GET /metrics: (98.765996ms) 200 [[Go-http-client/1.1] 172.17.0.162:49202]
Sep 04 06:25:24 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:24.194371 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/05c2105e-4b6a-11e7-ad4d-fa163ed46388-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "05c2105e-4b6a-11e7-ad4d-fa163ed46388" (UID: "05c2105e-4b6a-11e7-ad4d-fa163ed46388").
Sep 04 06:25:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:33.738025 16005 server.go:778] GET /metrics: (109.050426ms) 200 [[Go-http-client/1.1] 172.17.0.162:49224]
Sep 04 06:25:36 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:36.226919 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fccdbfd7-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fccdbfd7-360d-11e7-a88c-fa163ed46388" (UID: "fccdbfd7-360d-11e7-a88c-fa163ed46388").
Sep 04 06:25:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:43.737571 16005 server.go:778] GET /metrics: (108.474848ms) 200 [[Go-http-client/1.1] 172.17.0.162:49246]
Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.903049 16005 image_gc_manager.go:270] [imageGCManager]: Disk usage on "/dev/vda1" (/var/lib/docker/overlay) is at 94% which is over the high threshold (90%). Trying to free 7021084672 bytes
Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.916512 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:573ddcad1ff5243330f172efbf5d44a1c2c35abcae3c2839673817dff41764a6" to free 217108251 bytes
Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.918875 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:54e0498c3dadcfc3036086232145c96d334295a318c12cf8af025bf59ccd7d4e" to free 987088941 bytes
Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.922480 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:c1951085bcc973dcf859ca1842d049c2e394092c06a0dc8d5aa47f4b8b55de80" to free 42634444 bytes
Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.924666 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:958a7ae9e56979be256796dabd5845c704f784cd422734184999cf91f24c2547" to free 109368581 bytes
Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: E0904 06:25:45.927024 16005 kubelet.go:1165] Image garbage collection failed: wanted to free 7021084672, but freed 0 space with errors in image deletion: [Error response from daemon: {"message":"conflict: unable to delete 573ddcad1ff5 (cannot be forced) - image is being used by running container a0abb2be0dfd"}, Error response from daemon: {"message":"conflict: unable to delete 54e0498c3dad (must be forced) - image is being used by stopped container 675afc7fd4b2"}, Error response from daemon: {"message":"conflict: unable to delete c1951085bcc9 (cannot be forced) - image is being used by running container 0ce4eb29420d"}, Error response from daemon: {"message":"conflict: unable to delete 958a7ae9e569 (must be forced) - image is referenced in multiple repositories"}]
Sep 04 06:25:47 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:47.257883 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f1b4decb-33af-11e7-a587-fa163ed46388-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "f1b4decb-33af-11e7-a587-fa163ed46388" (UID: "f1b4decb-33af-11e7-a587-fa163ed46388").
Sep 04 06:25:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:53.733853 16005 server.go:778] GET /metrics: (104.78036ms) 200 [[Go-http-client/1.1] 172.17.0.162:49268]
Sep 04 06:25:57 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:57.186726 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77").
Sep 04 06:26:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:03.737476 16005 server.go:778] GET /metrics: (108.293704ms) 200 [[Go-http-client/1.1] 172.17.0.162:49290]
Sep 04 06:26:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:26:05.656712 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"}
Sep 04 06:26:06 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:06.213274 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77").
Sep 04 06:26:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:13.741089 16005 server.go:778] GET /metrics: (111.980692ms) 200 [[Go-http-client/1.1] 172.17.0.162:49312]
Sep 04 06:26:20 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:20.255745 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388").
Sep 04 06:26:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:23.734496 16005 server.go:778] GET /metrics: (102.819982ms) 200 [[Go-http-client/1.1] 172.17.0.162:49334]
Sep 04 06:26:24 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:24.274689 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77").
Sep 04 06:26:28 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:28.186208 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77").
Sep 04 06:26:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:33.200294 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77").
Sep 04 06:26:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:33.736186 16005 server.go:778] GET /metrics: (106.916768ms) 200 [[Go-http-client/1.1] 172.17.0.162:49356]
Sep 04 06:26:42 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:42.224907 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/05c2105e-4b6a-11e7-ad4d-fa163ed46388-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "05c2105e-4b6a-11e7-ad4d-fa163ed46388" (UID: "05c2105e-4b6a-11e7-ad4d-fa163ed46388").
Sep 04 06:26:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:43.738326 16005 server.go:778] GET /metrics: (109.178882ms) 200 [[Go-http-client/1.1] 172.17.0.162:49378]
Sep 04 06:26:46 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:46.235835 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fccdbfd7-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fccdbfd7-360d-11e7-a88c-fa163ed46388" (UID: "fccdbfd7-360d-11e7-a88c-fa163ed46388").
Sep 04 06:26:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:53.734508 16005 server.go:778] GET /metrics: (105.402837ms) 200 [[Go-http-client/1.1] 172.17.0.162:49400]
Sep 04 06:27:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:03.741988 16005 server.go:778] GET /metrics: (112.954759ms) 200 [[Go-http-client/1.1] 172.17.0.162:49422]
Sep 04 06:27:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:27:05.784294 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"}
Sep 04 06:27:09 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:09.205529 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77").
Sep 04 06:27:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:13.739225 16005 server.go:778] GET /metrics: (109.968574ms) 200 [[Go-http-client/1.1] 172.17.0.162:49444]
Sep 04 06:27:17 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:17.227946 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77").
Sep 04 06:27:17 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:17.228615 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f1b4decb-33af-11e7-a587-fa163ed46388-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "f1b4decb-33af-11e7-a587-fa163ed46388" (UID: "f1b4decb-33af-11e7-a587-fa163ed46388").
Sep 04 06:27:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:23.729005 16005 server.go:778] GET /metrics: (99.830606ms) 200 [[Go-http-client/1.1] 172.17.0.162:49466]
Sep 04 06:27:25 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:25.252704 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77").
Sep 04 06:27:27 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:27.258375 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388").
Sep 04 06:27:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:33.737406 16005 server.go:778] GET /metrics: (107.868122ms) 200 [[Go-http-client/1.1] 172.17.0.162:49488]
Sep 04 06:27:40 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:40.196971 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77").
Sep 04 06:27:40 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:40.197045 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77").
Sep 04 06:27:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:43.737770 16005 server.go:778] GET /metrics: (108.444824ms) 200 [[Go-http-client/1.1] 172.17.0.162:49510]
Sep 04 06:27:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:53.728799 16005 server.go:778] GET /metrics: (99.267527ms) 200 [[Go-http-client/1.1] 172.17.0.162:49532]
