Kubernetes Failed to Become Leader

Hi,

I am trying to deploy Singlestore on Kubernetes but the operator pod is failing with a “Failed to become leader” message. I have pasted the logs below, thanks in advance for any tips:

kubectl logs memsql-operator-548689648-7wb58
2021/03/31 21:23:43 main.go:55 {cmd} Go Version: go1.15.2
2021/03/31 21:23:43 main.go:56 {cmd} Go OS/Arch: linux/amd64
2021/03/31 21:23:43 main.go:57 {cmd} Version of operator-sdk: v0.2.1
2021/03/31 21:23:43 main.go:58 {cmd} Commit Hash: 8a8eb21f
2021/03/31 21:23:43 main.go:62 {cmd} Options:
2021/03/31 21:23:43 main.go:63 {cmd} --cores-per-unit: 8.000000
2021/03/31 21:23:43 main.go:64 {cmd} --memory-per-unit: 32.000000
2021/03/31 21:23:43 main.go:65 {cmd} --overpack-factor: 0.000000
2021/03/31 21:23:43 main.go:66 {cmd} --extra-cidrs: []
2021/03/31 21:23:43 main.go:67 {cmd} --external-dns-domain-name: {false }
2021/03/31 21:23:43 main.go:68 {cmd} --external-dns-ttl: {false 0}
2021/03/31 21:23:43 main.go:69 {cmd} --ssl-secret-name:
2021/03/31 21:23:43 main.go:70 {cmd} --merge-service-annotations: true
2021/03/31 21:23:43 main.go:93 {cmd} --backup-default-deadline-seconds: 3600
2021/03/31 21:23:43 main.go:101 {cmd} --backup-incremental-default-deadline-seconds: 3600
2021/03/31 21:23:43 main.go:125 {cmd} --fs-group-id: 5555
2021/03/31 21:23:43 leader.go:55 {leader} Trying to become the leader.
2021/03/31 21:23:43 leader.go:158 {leader} found namespace Namespace: “default”
2021/03/31 21:23:44 main.go:48 {cmd} Failed to become leader error: “Get “https://10.96.0.1:443/api?timeout=32s”: dial tcp 10.96.0.1:443: connect: no route to host”

Hi benji,

Generally, the Failed to become leader error will occur when there are two Operator instances trying to manage the same SingleStore cluster deployment. One Operator instance is already the leader, and the second instance is trying to become leader but is unable to (because there’s already a leader.)

To further help diagnose the problem, can you capture the output of pods running in the SingleStore namespace? (From our error output it looks like they are running in the default namespace.)

Are you running your Kubernetes cluster on a CSP provider like AWS or Google Cloud? Can you provide more information on the Kubernetes cluster environment?

It would also be helpful to verify your deployment configs, if you are able to share them then please DM them directly to me to review. The files that would be helpful are the Operator deployment and the MemSQL Cluster YAML config files.

Kind regards,
Cindy