Kubernetes Memsql cluster

Hello,
I am using kubernetes 13.5 and rook/ceph 0.93
when running on a single machine everything works OK.
when using a cluster I get the following issue,

Unable to mount volumes for pod “node-memsql-cluster-leaf-ag1-0_default(65b25f7c-77ca-11e9-b0df-42010a840042)”: timeout expired waiting for volumes to attach or mount for pod “default”/“node-memsql-cluster-leaf-ag1-0”. list of unmounted volumes=[pv-storage]. list of unattached volumes=[pv-storage probes default-token-jfrf5]

Please note that Ceph creates the SC, VPC and VP the way it should,

root@memsql-aio-1:~/3rd-party/memsql# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-65af3c11-77ca-11e9-b0df-42010a840042 256Gi RWO Delete Bound default/pv-storage-node-memsql-cluster-master-0 rook-ceph-block 73m
pvc-65b1eefe-77ca-11e9-b0df-42010a840042 1Ti RWO Delete Bound default/pv-storage-node-memsql-cluster-leaf-ag1-0 rook-ceph-block 73m

root@memsql-aio-1:~/3rd-party/memsql# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-default-consul-server-0 Bound pvc-a2e662f2-77b0-11e9-b0df-42010a840042 10Gi RWO local-path 4h18m
pv-storage-node-memsql-cluster-leaf-ag1-0 Bound pvc-65b1eefe-77ca-11e9-b0df-42010a840042 1Ti RWO rook-ceph-block 73m

NAME PROVISIONER AGE
local-path rancher.io/local-path 4h18m
rook-ceph-block ceph.rook.io/block 3h58m

root@memsql-aio-1:~/3rd-party/memsql# cat memsql-cluster.yaml
apiVersion: memsql.com/v1alpha1
kind: MemsqlCluster
metadata:
name: memsql-cluster
spec:
license: …
adminHashedPassword: …
releaseID: 563337fd-bbab-4f7d-aefe-650675e3b1b8
redundancyLevel: 1

aggregatorSpec:
count: 1
height: 0.5
storageGB: 256
storageClass: rook-ceph-block

leafSpec:
count: 1
height: 1
storageGB: 1024
storageClass: rook-ceph-block
root@memsql-aio-1:~/3rd-party/memsql#

Thanks so much,

Meir

Hello,

Could you please help us with this issue?

Thanks so much

Hi Meir, it would be helpful to have some more information about the pods in the cluster. Can you run “kubectl describe pods” and “kubectl describe pvc”?