High availability on two hosts

Help
1

Hi, the documentation of memsql 7 has an option to deploy 2 hosts 4 nodes cluster though I have a couple of things that are missing:

  1. I see there is one leaf that is also the Master, can I make the other leaf the child ? I dont see the role in the config
  2. Assuming #1 is possible, why is there no “high_availability: true” ? If there are 2 leaves with replication factor of 2 and 2 aggregators then this should be a high availability solution

Thanks !

2

Hi @elad,

Thank you for using MemSQL. Would you please tell me more about your configuration? I would like to see your result of this command “memsql-admin list-nodes”.

This link has some good information about managing HA. Just in case you did not have a chance to go through it yet: SingleStoreDB Cloud · SingleStore Documentation

I am looking forward to learning the details of your configuration.

3

Hi @ywang

Right now we have a HA cluster of 4 servers: 1 master aggregator, 1 child aggregator and 2 leaves

I want to reduce the amount of servers to 2 but keeping the HA. My idea was to make each leaf an aggregator

I was wondering if it’s possible to do ? Will it still be HA ?

Thank you,
Elad

4

Good morning, @elad,

Yes a configuration of 2 hosts with MA, CA and two leaves is supported.

I tried exactly the same configuration like yours with 2 aggregators and 2 leaves. I created one additional node on each of the MA and C. Then I added the new nodes as leaf nodes. I ran REBALANCE PARTITIONS then remove the old leaf nodes. Now I have two hosts, one with the MA and one leaf, the other has the CA and another leaf. This is my test environment so I didn’t care about my data that much. Please make good backups if you want to keep your existing data.

To test my HA is still there, I remove one the leaf in my MA. I can still query all data of my database.

In MemSQL Studio window, you can see the topology of your nodes like this.

Screen Shot 2020-05-04 at 10.46.51 AM
Screen Shot 2020-05-04 at 10.46.51 AM1484×577 53 KB

You can also run “SHOW CLUSTER STATUS;” to see more information about your cluster, DBs and DB partitions.

5

That is great news !!! thank you !

6

Hi @ywang I am trying to do this configuration and its not working on my end. I have right now a 4 nodes cluster running on 4 hosts, here is the list-nodes:
±-----------±-----------±----------------------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
±-----------±-----------±----------------------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| 3EFC99AACD | Master | 10.1.2.5 | 3306 | Running | True | 7.0.16 | Online | | 0.0.0.0 |
| B79F944FDD | Aggregator | memsql-agg-01.server.local | 3306 | Running | True | 7.0.16 | Online | | 0.0.0.0 |
| A337CB882E | Leaf | memsql-leaf-01.server.local | 3306 | Running | True | 7.0.16 | Online | 1 | 0.0.0.0 |
| A805572DF1 | Leaf | memsql-leaf-02.server.local | 3306 | Running | True | 7.0.16 | Online | 2 | 0.0.0.0 |
±-----------±-----------±--------------------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+

I want to take one of the leaves and give it another role to be an aggregator. When I run the add-aggregator command I get the error:
stderr: Node already has role Leaf

How can it be done ?

7

Hi @elad,

Glad to see you here again. Have you try these steps yet? As an example, A337CB882E on memsql-leaf-01.server.local:3306 is the one that you want to make it as an aggregator.

memsql-admin remove-leaf --memsql-id A337CB882E
memsql-admin add-aggregator --memsql-id A337CB882E

Please let me know if this helps.

Best,

Yu-wang

8

@ywang I don’t want to remove this leaf. I want the server to be both a lead and child aggregator

9

You do need to use a different port number for the aggregator node on the same host since the original port number 3306 is used as a leaf node for that host. I suspect that is the reason. If this is not the reason, can you send me the exact command you used to add the aggregator?

10

@ywang Yes it might be the reason, I am running this command: memsql-admin add-aggregator --memsql-id A805572DF1

But according to what you say do I need to create another node ? or I just add-aggregator with a different port ? The error I get is saying that this node already has a role, its not telling me anything about the port

11

A node and setting its role are two different things. So yes, you do need to create a new node on a different port, then add it as a leaf or an aggregator.

This is what I did when I tested your scenario. Let’s say I have MA, CA, LEAF1 and LEAF2 already on 4 hosts.

  1. Create a new node LEAF-on-MA on the MA host + different port
  2. Add the new node as a LEAF node
  3. Remove LEAF1 and rebalance. Now I have MA, LEAF-on_MA, CA and LEAF2.

Then I do the same steps to add LEAF-on-CA. At the end I have MA, LEAF-on_MA on one host. CA and LEAF-on-CA on another host with all the partitions of my db.

I am sure there are other combinations of steps that can get you to the same steps.

Hope this helps,

Yu-wang

12

Ok, the server has a mount with the data dir in a different location. When I installed the node I used the --datadir argument. Do I need to indicate it here as well ? Wouldnt it overwrite the data directory ?

13

Sorry for the late response. Yes you can use:
memsql-admin create-node … --datadir …

By the way, would you please tell me which company you work for?

Thanks,

Yu-wang

14

I was wondering if it’s possible to do?