Unable to add leaves to the cluster - fresh install

Help
1

I’m doing a fresh installation of MemSQL version 6.7.14 over Azure memory optimized Ubuntu 16.04 VMs

The problem is that I cant’ add any leaves to the cluster.

The error message:

memsql-admin add-leaf --memsql-id FB18A9EEF13E105EFE7A1B9D54184B6D9176D2DC --password ‘’
Toolbox will perform the following actions on host 172.16.0.6:
_ · Run ‘memsqlctl add-leaf --host 172.16.0.10 --port 3306 --user root --password ●●●●●●’_

Would you like to continue? [y/N]: y
✘ Failed to run ‘memsqlctl add-leaf’
error running memsqlctl: error running command: "/usr/bin/memsqlctl" "--json" "--yes" "add-leaf" "--host" "172.16.0.10" "--port" "3306" "--user" "root" "--secure-password" "<secure_pass>": exit status 1
stderr: Error 1045: Access denied for user ‘root’@‘172.16.0.6’ (using password: YES)
Could not add leaf to cluster. Verify network connectivity between the node at 172.16.0.10:3306 and the master aggregator.
Please ensure that you have entered the correct password for the node.
If the node does not yet exist, run ‘memsql-admin create-node’ and then retry this command.

Debug and Context:
Infra:
1 master agg
1 child agg
10 leaves

The installation is being done according to this manual:

On each host I followed the OS recommendations here:

On each leaf I have 3 disk that act as a RAID cluster

I noticed that the memsql user in /etc/passwd is set to /bin/false ( I changed it to /bin/bash but didn’t help).

I tried running the command memsqlctl add-leaf --host 172.16.0.10 --port 3306 --user root --password ●●●●●●’_
manually but got the same results both as working user and as memsql user.
I can ssh from the master to the leaf
Both hosts are on the same Azure vnet which means all ports are still open.

listing the nodes I see my intended leaf registered:
memsql-admin list-nodes
±-----------±-----------±------------±-----±--------------±-------------±--------±---------------±-------------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group |
±-----------±-----------±------------±-----±--------------±-------------±--------±---------------±-------------------+
| ABBFA82138 | Master | 172.16.0.6 | 3306 | Running | True | 6.7.14 | Online | |
| 8D7FE2C60C | Aggregator | 172.16.0.14 | 3306 | Running | True | 6.7.14 | Online | |
| FB18A9EEF1 | Unknown | 172.16.0.10 | 3306 | Running | True | 6.7.14 | Online | |
±-----------±-----------±------------±-----±--------------±-------------±--------±---------------±-------------------+

What am I missing?
Any suggestions or workaround that will get us there would be greatly appreciated.

Regards,
Naim

2

Hello Naim,
Unfortunately this sounds like the Master Aggregator is not able to connect to the leaf node over port 3306. Can you connect to your master aggregator using the MemSQL client (apt install memsql-client if you have our debian repo installed) and run ADD LEAF 'root'@'172.16.0.10':3306? If that gives you an error then the issue is 100% network related. Please report back and I will continue to help you through this issue.

Thanks!

3

Hi Carl, Facing the same issue. What can be the resolution?

4

Hi Harsh! :wave: Welcome and thanks for your question. Sorry you are having this experience.
Can you specify what version you are running?
And the error message you are getting?