Can't Add Leaf to Free Version

Hi There,

I am currently running a cluster on the free version with one aggregator node and two leaf nodes. I have tried to add another leaf node but it fails with error message

Leaf node could not be added because you are using the MemSQL free license which has a limit of 4 license units and after adding the node you would be using 6 license units

Here is the output from sdb-admin list-nodes:

±-----------±--------±-------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
±-----------±--------±-------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| A9D5B0E9D7 | Master | 192.168.1.33 | 3306 | Running | True | 7.3.6 | Online | | 0.0.0.0 |
| B3D5677F16 | Leaf | 192.168.1.34 | 3306 | Running | True | 7.3.6 | Online | 1 | 0.0.0.0 |
| 1D72FECE5A | Leaf | 192.168.1.36 | 3306 | Running | True | 7.3.6 | Online | 2 | 0.0.0.0 |
| 45D87932ED | Unknown | 192.168.1.37 | 3306 | Running | True | 7.3.6 | Online | | 0.0.0.0 |
±-----------±--------±-------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+

Output from sdb-toolbox-config list-hosts

±-------------±-----------±-------------±------------------+
| Host | Local Host | SSH address | Identity File |
±-------------±-----------±-------------±------------------+
| 192.168.1.33 | No | 192.168.1.33 | /root/.ssh/id_rsa |
| 192.168.1.34 | No | 192.168.1.34 | /root/.ssh/id_rsa |
| 192.168.1.36 | No | 192.168.1.36 | /root/.ssh/id_rsa |
| 192.168.1.35 | No | 192.168.1.35 | /root/.ssh/id_rsa |
| 192.168.1.37 | No | 192.168.1.37 | /root/.ssh/id_rsa |
±-------------±-----------±-------------±------------------+

Output from cat ~/memsql/nodes/nodes.hcl looks similar on all hosts:

version = 1
node {
memsqlConfigPath = “/root/memsql/nodes/a3d9e45f-b19e-4f0d-ac8a-004a640eae53/memsql.cnf”
password {
cipherText = “xxxxxxxxx”
}
}
secure = “xxxxxxxxx”

Output from ./sdb-admin add-leaf --memsql-id 45D87932EDCC5DB4D66420067801D96579A56E80 --password xxxxxxx

root@192.168.1.33’s password:
failed to find host with master: no master aggregator found in this cluster
[root@hadoop-namenode singlestoredb-toolbox-1.11.0-64ef3b1cd1]# ./sdb-admin add-leaf --memsql-id 45D87932EDCC5DB4D66420067801D96579A56E80 --password Monday@01 -vvv
2021/03/24 11:25:06 logging.go:37 Running command #11 /usr/bin/printenv SUDO_UID
2021/03/24 11:25:06 logging.go:52 Command #11 failed with exit code 1 in 789.664µs
Debug 2021/03/24 11:25:06 env.go:486 cannot resolve runtime dir: XDG_RUNTIME_DIR not set and no override specified
2021/03/24 11:25:06 backup_cache.go:43 Could not load backup cache from /root/.local/share/singlestoredb-toolbox/backup-cache.hcl: could not read file /root/.local/share/singlestoredb-toolbox/backup-cache.hcl: open /root/.local/share/singlestoredb-toolbox/backup-cache.hcl: no such file or directory
2021/03/24 11:25:06 env.go:153 Toolbox version: 1.11.0
2021/03/24 11:25:06 env.go:154 Toolbox env: &{ConfigPath:/root/.config/singlestoredb-toolbox/toolbox.hcl StateFilePath:/root/.local/share/singlestoredb-toolbox/toolbox-state.hcl CacheFilePath:/root/.local/share/singlestoredb-toolbox/toolbox-cache.hcl BackupCacheFilePath:/root/.local/share/singlestoredb-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/root/.cache/singlestoredb-toolbox/restore.lock SessionDir:/tmp/singlestoredb-toolbox102438225 SSHBinaryPath:/usr/bin/ssh}
2021/03/24 11:25:06 logging.go:37 Running command #12 /usr/bin/ssh -oControlMaster=yes -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket -N -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oBatchMode=no -oIdentityFile=/root/.ssh/id_rsa 192.168.1.33
root@192.168.1.33’s password:
2021/03/24 11:25:36 connection_manager.go:156 Testing SSH configuration for 192.168.1.33
2021/03/24 11:25:36 logging.go:37 Running command #13 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 echo ping
2021/03/24 11:25:36 logging.go:52 Command #13 took 180.376385ms
2021/03/24 11:25:36 logging.go:37 Running command #14 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 id -u
2021/03/24 11:25:36 logging.go:52 Command #14 took 24.147012ms
2021/03/24 11:25:36 logging.go:37 Running command #15 /usr/bin/ssh -oControlMaster=yes -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket -N -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oBatchMode=no -oIdentityFile=/root/.ssh/id_rsa 192.168.1.34
2021/03/24 11:25:36 connection_manager.go:156 Testing SSH configuration for 192.168.1.34
2021/03/24 11:25:36 logging.go:37 Running command #16 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket 192.168.1.34 echo ping
2021/03/24 11:25:36 logging.go:52 Command #16 took 265.579986ms
2021/03/24 11:25:36 logging.go:37 Running command #17 /usr/bin/ssh -oControlMaster=yes -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket -N -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oBatchMode=no -oIdentityFile=/root/.ssh/id_rsa 192.168.1.36
2021/03/24 11:25:36 connection_manager.go:156 Testing SSH configuration for 192.168.1.36
2021/03/24 11:25:36 logging.go:37 Running command #18 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket 192.168.1.36 echo ping
2021/03/24 11:25:37 logging.go:52 Command #18 took 237.446823ms
2021/03/24 11:25:37 logging.go:37 Running command #19 /usr/bin/ssh -oControlMaster=yes -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket -N -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oBatchMode=no -oIdentityFile=/root/.ssh/id_rsa 192.168.1.35
2021/03/24 11:25:37 connection_manager.go:156 Testing SSH configuration for 192.168.1.35
2021/03/24 11:25:37 logging.go:37 Running command #20 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket 192.168.1.35 echo ping
2021/03/24 11:25:37 logging.go:52 Command #20 took 218.894553ms
2021/03/24 11:25:37 logging.go:37 Running command #21 /usr/bin/ssh -oControlMaster=yes -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket -N -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oBatchMode=no -oIdentityFile=/root/.ssh/id_rsa 192.168.1.37
2021/03/24 11:25:37 connection_manager.go:156 Testing SSH configuration for 192.168.1.37
2021/03/24 11:25:37 logging.go:37 Running command #22 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 echo ping
2021/03/24 11:25:37 logging.go:52 Command #22 took 110.461777ms
2021/03/24 11:25:37 logging.go:37 Running command #23 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 id -u
2021/03/24 11:25:37 logging.go:52 Command #23 took 15.137758ms
2021/03/24 11:25:37 logging.go:37 Running command #24 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:37 logging.go:37 Running command #25 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 test -e /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:37 Running command #26 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket 192.168.1.34 id -u
2021/03/24 11:25:37 logging.go:52 Command #24 took 13.758676ms
2021/03/24 11:25:37 logging.go:37 Running command #27 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 cat /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #25 took 22.764041ms
2021/03/24 11:25:37 logging.go:37 Running command #28 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 cat /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #27 took 13.975481ms
2021/03/24 11:25:37 logging.go:37 Running command #29 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #26 took 28.357506ms
2021/03/24 11:25:37 logging.go:37 Running command #30 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket 192.168.1.34 test -e /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:37 Running command #31 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket 192.168.1.36 id -u
2021/03/24 11:25:37 logging.go:52 Command #29 took 13.915629ms
2021/03/24 11:25:37 logging.go:37 Running command #32 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl list-nodes
2021/03/24 11:25:37 logging.go:52 Command #28 took 22.520407ms
2021/03/24 11:25:37 logging.go:37 Running command #33 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 test -e /root/memsql/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #30 took 24.224243ms
2021/03/24 11:25:37 logging.go:37 Running command #34 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket 192.168.1.34 cat /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #31 took 24.553545ms
2021/03/24 11:25:37 logging.go:37 Running command #35 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket 192.168.1.36 test -e /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:37 Running command #36 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket 192.168.1.35 id -u
2021/03/24 11:25:37 logging.go:52 Command #33 took 22.016602ms
2021/03/24 11:25:37 logging.go:37 Running command #37 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl list-nodes
2021/03/24 11:25:37 logging.go:52 Command #35 took 23.048203ms
2021/03/24 11:25:37 logging.go:37 Running command #38 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket 192.168.1.36 cat /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #34 took 24.803486ms
2021/03/24 11:25:37 logging.go:37 Running command #39 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket 192.168.1.34 test -e /root/memsql/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #36 took 26.224658ms
2021/03/24 11:25:37 logging.go:37 Running command #40 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket 192.168.1.35 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #38 took 23.749829ms
2021/03/24 11:25:37 logging.go:37 Running command #41 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket 192.168.1.36 test -e /root/memsql/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #39 took 23.958962ms
2021/03/24 11:25:37 logging.go:37 Running command #42 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/2.socket 192.168.1.34 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl list-nodes
2021/03/24 11:25:37 logging.go:52 Command #40 took 24.785003ms
2021/03/24 11:25:37 logging.go:37 Running command #43 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket 192.168.1.35 cat /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #32 took 64.96846ms
2021/03/24 11:25:37 logging.go:52 Command #41 took 22.221352ms
2021/03/24 11:25:37 logging.go:37 Running command #44 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/3.socket 192.168.1.36 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl list-nodes
2021/03/24 11:25:37 logging.go:52 Command #43 took 24.684505ms
2021/03/24 11:25:37 logging.go:37 Running command #45 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket 192.168.1.35 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #37 took 77.972957ms
2021/03/24 11:25:37 logging.go:52 Command #45 took 24.064328ms
2021/03/24 11:25:37 logging.go:37 Running command #46 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/4.socket 192.168.1.35 /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl list-nodes
2021/03/24 11:25:37 logging.go:52 Command #46 took 32.13505ms
2021/03/24 11:25:37 logging.go:52 Command #42 took 90.130195ms
2021/03/24 11:25:37 logging.go:52 Command #44 took 84.049231ms
2021/03/24 11:25:37 logging.go:37 Running command #47 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 id -u
2021/03/24 11:25:37 logging.go:52 Command #47 took 15.088897ms
2021/03/24 11:25:37 logging.go:37 Running command #48 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #48 took 13.347398ms
2021/03/24 11:25:37 logging.go:37 Running command #49 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 cat /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #49 took 13.400716ms
2021/03/24 11:25:37 logging.go:37 Running command #50 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #50 took 13.034482ms
2021/03/24 11:25:37 logging.go:37 Running command #51 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl describe-node --memsql-id 45D87932EDCC5DB4D66420067801D96579A56E80 --property Port
2021/03/24 11:25:37 logging.go:52 Command #51 took 54.271108ms
2021/03/24 11:25:37 logging.go:37 Running command #52 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 id -u
2021/03/24 11:25:37 logging.go:52 Command #52 took 23.159469ms
2021/03/24 11:25:37 logging.go:37 Running command #53 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 test -e /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #53 took 20.303848ms
2021/03/24 11:25:37 logging.go:37 Running command #54 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 cat /root/memsql/packages.hcl
2021/03/24 11:25:37 logging.go:52 Command #54 took 21.585537ms
2021/03/24 11:25:37 logging.go:37 Running command #55 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 test -e /root/memsql/memsqlctl.hcl
2021/03/24 11:25:37 logging.go:52 Command #55 took 21.36434ms
2021/03/24 11:25:37 logging.go:37 Running command #56 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl list-nodes --role Master
2021/03/24 11:25:38 logging.go:52 Command #56 took 74.960062ms
2021/03/24 11:25:38 logging.go:37 Running command #57 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 id -u
2021/03/24 11:25:38 logging.go:52 Command #57 took 14.746695ms
2021/03/24 11:25:38 logging.go:37 Running command #58 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:38 logging.go:52 Command #58 took 13.523279ms
2021/03/24 11:25:38 logging.go:37 Running command #59 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 cat /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/packages.hcl
2021/03/24 11:25:38 logging.go:52 Command #59 took 13.935745ms
2021/03/24 11:25:38 logging.go:37 Running command #60 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 test -e /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl
2021/03/24 11:25:38 logging.go:52 Command #60 took 12.863315ms
2021/03/24 11:25:38 logging.go:37 Running command #61 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/5.socket 192.168.1.37 /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /opt/singlestore/memsql-server-7.3.6-ae5d1857b8/memsqlctl.hcl list-nodes
2021/03/24 11:25:38 logging.go:52 Command #61 took 61.705816ms
Toolbox will perform the following actions on host 192.168.1.33:
· Run ‘memsqlctl add-leaf --host 192.168.1.37 --port 3306 --user root --password ●●●●●●’
Would you like to continue? [y/N]: y
2021/03/24 11:26:11 logging.go:37 Running command #62 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl print-secure-key
2021/03/24 11:26:11 logging.go:52 Command #62 took 74.266902ms
2021/03/24 11:26:11 logging.go:37 Running command #63 /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl add-leaf --host 192.168.1.37 --port 3306 --user root --secure-password '7N3eRkcjxNtXf+KLpasIycVup3eORlxNc8TvVaXpV/50xI20Ag=='
2021/03/24 11:26:11 logging.go:52 Command #63 failed with exit code 1 in 144.10461ms
2021/03/24 11:26:11 client.go:114 memsqlctl error output:
stdout: Error 2374: Leaf node could not be added because you are using the MemSQL free license which has a limit of 4 license units and after adding the node you would be using 6 license units. To purchase more capacity, contact team@memsql.com. See Capacity Limit Errors · SingleStore Documentation for instructions on how to create a cluster with up to 4 license units.
stderr: (empty)
✘ Failed to run ‘memsqlctl add-leaf’
2021/03/24 11:26:11 logging.go:52 Command #21 took 34.232285837s
2021/03/24 11:26:11 logging.go:52 Command #12 took 1m5.082493082s
2021/03/24 11:26:11 logging.go:52 Command #15 took 35.200417658s
2021/03/24 11:26:11 logging.go:52 Command #17 took 34.85129886s
2021/03/24 11:26:11 logging.go:52 Command #19 took 34.535040699s
error running memsqlctl: error running command: /usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/singlestoredb-toolbox102438225/1.socket 192.168.1.33 /root/memsql/memsql-server-7.3.6-ae5d1857b8/memsqlctl --json --yes --config /root/memsql/memsqlctl.hcl add-leaf --host 192.168.1.37 --port 3306 --user root --secure-password '7N3eRkcjxNtXf+KLpasIycVup3eORlxNc8TvVaXpV/50xI20Ag==': exit status 1:
stdout: Error 2374: Leaf node could not be added because you are using the MemSQL free license which has a limit of 4 license units and after adding the node you would be using 6 license units. To purchase more capacity, contact team@memsql.com. See Capacity Limit Errors · SingleStore Documentation for instructions on how to create a cluster with up to 4 license units.
stderr: (empty)
Failed to add the leaf node to the cluster. Verify network connectivity between the node at 192.168.1.37:3306 and the Master Aggregator node.
Please ensure that you have entered the correct password for the node.
If the node does not yet exist, run ‘sdb-admin create-node’ and then retry this command.

Beniamin.

Thank you for providing all of this information.

I want to rule out one last high-level thing; can you provide the core count and memory specs of your hosts?

In our email exchange, I did not mention that our node unit count is based on 4 leaf nodes of 32 GB each (for a total of 128 GBs). Also, a node can have up to 8vCPU. If you exceed these limits per node, it will count as additional nodes which could explain the discrepancy.

Let me know the specs and if this is not the case. In parallel, we will also investigate the data you provided further.

Hi Roxanna,

Thanks for your response. Each host has 65GB ram and 16 cores, so I guess that explains the result? Is there any way in which I can configure each node to use less cores and cores?

Cheers
Benji

Thank you for confirming. I agree that this explains the results. Since the hosts have 65GB of RAM and 16 cores, they are getting counted as more than 1 node.

I’m not aware of a way you can configure each node to use less cores and RAM. You will have to provision different machines for the installation.

If you are looking to trial our product with larger machines, we recommend you sign up and use our managed service. You get 250 free credits and you can sign up on-demand as well. The managed service is also seamless because installation is one-click, and we manage all the physical aspects of the deployment.

I ran into this same problem, and had to deduce your error message to the fact I was trying to run on hardware greater than the license provided.

Your instructions simply need ONE small modification.

sudo docker run -i --init --cpus 4 --memory=16gb --name singlestore-ciab -e LICENSE_KEY=“$LICENSE” -e ROOT_PASSWORD=“$PASSWORD” -p 3306:3306 -p 8080:8080 singlestore/cluster-in-a-box

Limit the cpu cores and memory, and your docker setup works. In fact, it should have auto-detected the limit from the license and did the right thing from the start. I’ve re-written your golang code before while I was at IEX, I can do it again :slight_smile:

@ryan.dietrich thanks for pointing this out! What instructions specifically are you talking about? Could you give the URL(s) of the page(s) you were looking at? That’d help us localize where to change things.