SingleStore operator not creating cluster

Help
1

I’m following the docs here to spin up a cluster to evaluate on AWS EKS: Create the Object Definition Files · SingleStore Documentation

And was able to create the operator with kubectl create -f sdb-operator.yaml
But when I run kubectl create -f sdb-cluster.yaml, the operator never creates the cluster pods. Checking the operator logs, it doesn’t look like it’s doing anything. This is all that’s in the log even after applying the cluster file:

2022/09/20 14:55:56 deleg.go:121	{cmd}	Go Version: go1.18.2	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Go OS/Arch: linux/amd64	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Operator Version: 3.0.33	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Commit Hash: db8f5aff	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Build Time: 2022-09-08T14:43:05Z	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Options:	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--cores-per-unit: 8.000000	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--memory-per-unit: 32.000000	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--overpack-factor: 0.000000	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--extra-cidrs: []	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--external-dns-domain-name: {false }	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--external-dns-ttl: {false 0}	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--ssl-secret-name: 	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--merge-service-annotations: true	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--backup-default-deadline-seconds: 3600	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--backup-incremental-default-deadline-seconds: 3600	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--cluster-id: singlestore-reporting-uat	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--fs-group-id: 5555	
I0920 14:55:57.699056       1 request.go:665] Waited for 1.041794107s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/batch/v1beta1?timeout=6m0s
2022/09/20 14:55:58 deleg.go:121	{controller-runtime.metrics}	Metrics server is starting to listen	addr: "0.0.0.0:9090"  
2022/09/20 14:55:58 deleg.go:121	{cmd}	starting manager	
2022/09/20 14:55:58 logr.go:249	Starting server	addr: "[::]:9090"  path: "/metrics"  kind: "metrics"  
I0920 14:55:58.464460       1 leaderelection.go:248] attempting to acquire leader lease default/memsql-operator-lock-singlestore-reporting-uat...
I0920 14:55:58.478257       1 leaderelection.go:258] successfully acquired lease default/memsql-operator-lock-singlestore-reporting-uat
2022/09/20 14:55:58 logr.go:249	{events}	Normal	object: "{Kind:ConfigMap Namespace:default Name:memsql-operator-lock-singlestore-reporting-uat UID:bff9f3a3-61c8-4349-a924-9664fafbdc95 APIVersion:v1 ResourceVersion:247815745 FieldPath:}"  reason: "LeaderElection"  message: "sdb-operator-7cc785cb69-lr8q2_b5e7e991-597c-4712-9f18-96a6d0db31f3 became leader"  
2022/09/20 14:55:58 logr.go:249	{events}	Normal	object: "{Kind:Lease Namespace:default Name:memsql-operator-lock-singlestore-reporting-uat UID:e47d54f4-c608-4423-9944-f1a1e8724210 APIVersion:coordination.k8s.io/v1 ResourceVersion:247815746 FieldPath:}"  reason: "LeaderElection"  message: "sdb-operator-7cc785cb69-lr8q2_b5e7e991-597c-4712-9f18-96a6d0db31f3 became leader"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1alpha1.MemsqlCluster"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1alpha1.MemsqlCluster"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1.StatefulSet"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1.Service"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1.Secret"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting Controller	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting workers	reconciler kind: "MemsqlCluster"  reconciler group: "memsql.com"  worker count: "1"

This is my sdb-cluster.yaml:

apiVersion: memsql.com/v1alpha1
kind: MemsqlCluster
metadata:
  name: sdb-cluster
spec:
  license: {license}
  adminHashedPassword: "*{hashed_password}"
  nodeImage:
    repository: memsql/node
    tag: alma-7.8.14-8b65b22706

  redundancyLevel: 1

  serviceSpec:
    objectMetaOverrides:
      labels:
        custom: label
      annotations:
        custom: annotations

  aggregatorSpec:
    count: 2
    height: 0.5
    storageGB: 256
    storageClass: ssd

    objectMetaOverrides:
      annotations:
        optional: annotation
      labels:
        optional: label

  leafSpec:
    count: 2
    height: 0.5
    storageGB: 1024
    storageClass: ssd

    objectMetaOverrides:
      annotations:
        optional: annotation
      labels:
        optional: label

And the sdb-operator.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sdb-operator
  labels:
    app.kubernetes.io/component: operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: sdb-operator
  template:
    metadata:
      labels:
        name: sdb-operator
    spec:
      serviceAccountName: sdb-operator
      containers:
        - name: sdb-operator
          image: memsql/operator:3.0.32-almalinux-db8f5aff
          imagePullPolicy: Always
          args: [
            # Cause the operator to merge rather than replace annotations on services
            "--merge-service-annotations",
            # Allow the process inside the container to have read/write access to the `/var/lib/memsql` volume.
            "--fs-group-id", "5555",
            "--cluster-id", "singlestore-reporting-uat"
          ]
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "sdb-operator"
2

Welcome aboard kryo! :wave:

Sorry to hear this issue is happening. Are you running on the managed or self-hosted service? What is your version number?

3

Thanks Maria.

I’m trying to set up a self-hosted cluster for evaluation in our kubernetes cluster.
Using the latest docker images available from dockerhub

SingleStore: memsql/node:alma-7.8.14-8b65b22706
Operator: memsql/operator:3.0.32-almalinux-db8f5aff

1 Like
4

Thanks for the info kryo. Could you try 2 things for me?

  1. Can you try to preload images here on the local container repo?
    – SingleStoreDB Cloud · SingleStore Documentation

  2. Can you confirm that in your attempts you followed the order outlined here?
    – Deploy a SingleStoreDB Cluster · SingleStore Documentation

Please keep us updated. :rocket:

5

the operator will filter custom resources by the --cluster-id argument. This is synonymous with the custom resource name. In your case you can change the cluster id in the operator to match your resource

"--cluster-id", "sdb-cluster"
1 Like
6

Thanks, bremy, that was it. I must have missed that. I swear I went over those docs 10 times trying to find what I missed, but as soon as I synced up the --cluster-id in sdb-operator.yaml and the name in sdb-cluster.yaml it picked it up.

1 Like
7

can someone tell in details, regarding changes needs to be done in sdb-cluster.yaml. I am facing same issue.

@bremy