Failed to find host with master: no master aggregator found in this cluster

adriant · August 4, 2021, 3:32pm

Hi everyone,

I’m following the SingleStore integration with Prometheus and Grafana tutorial, and I’m running into some issues on registering the master aggregator on the cluster.

I’m using the Docker version of the CIAB version 7.3.11 on Debian 10, together with other containers, triggered by Docker Compose

My initial command was (following the tutorial)

sdb-admin configure-monitoring --exporter-user root --exporter-password singlestore
with the error: failed to find host with master: no master aggregator found in this cluster

Then, I tried to register the master aggregator

db-admin register-node --host 172.18.0.2 --port 3306 --password singlestore --memsql-config /var/lib/memsql/d6f6f2a3-757e-4c72-afe9-ce776a0ff35c/memsql.cnf
with the error: no host found with hostname 172.18.0.2

Studio is running well on http://172.18.0.2:8080/ and if I trigger the Console commands, I see both master aggregator and leaf details

show aggregators;
±-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Host | Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | Master_Aggregator | NodeId |
±-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 127.0.0.1 | 3306 | online | 6 | NULL | 1 | 1 |
±-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

show leaves;
±--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Host | Port | Availability_Group | Pair_Host | Pair_Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | NodeId | Grace_Period_In_seconds |
±--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 127.0.0.1 | 3307 | 1 | NULL | NULL | online | 33 | 0.327 | 2 | NULL |
±--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

This is the content of both memsql.conf files for the Master Aggregator and the Leaf

cat /var/lib/memsql/d6f6f2a3-757e-4c72-afe9-ce776a0ff35c/memsql.cnf
[server]
auditlogsdir = auditlogs
bind_address = 0.0.0.0
datadir = data
pid_file = memsqld.pid
plancachedir = plancache
port = 3306
tracelogsdir = tracelogs
basedir = .
core_file = true
lc_messages_dir = ./share
lock_wait_timeout = 60
minimum_core_count = 0
minimum_memory_mb = 0
snapshot_trigger_size = 2g
socket = memsql.sock
tmpdir = .

cat /var/lib/memsql/7efc0c16-8c9c-4501-99a2-946f4318795f/memsql.cnf
[server]
auditlogsdir = auditlogs
bind_address = 0.0.0.0
datadir = data
pid_file = memsqld.pid
plancachedir = plancache
port = 3307
tracelogsdir = tracelogs
basedir = .
core_file = true
lc_messages_dir = ./share
lock_wait_timeout = 60
minimum_core_count = 0
minimum_memory_mb = 0
snapshot_trigger_size = 2g
socket = memsql.sock
tmpdir = .

My first thoughts, this issue might be from a network/ IP assignment issue.
The Docker IP address for the CIAB is 172.18.0.2, the Internal CIAB cluster is still on 127.0.0.1 and in the memsql.cnf files the bind_address are 0.0.0.0

Do I need to update the internal CIAB IPs to match the Docker IP address?

Please advise.

Thanks,
Adrian

adriant · August 4, 2021, 6:50pm

Hi again,

After few hours of try and fail, I found the solution below

memsql-toolbox-config register-host --cluster-hostname 172.18.0.2 --localhost --memsqlctl-config-path /var/lib/memsql/nodes.hcl
Toolbox is about to register the following host:
· Host: 172.18.0.2
· Localhost: true
· Memsqlctl Config Path: /var/lib/memsql/nodes.hcl

Would you like to continue? [y/N]: y
✓ Successfully registered host 172.18.0.2
±-----------±-----------±------------±--------------+
| Host | Local Host | SSH address | Identity File |
±-----------±-----------±------------±--------------+
| 172.18.0.2 | Yes | | |
±-----------±-----------±------------±--------------+

sdb-admin configure-monitoring --exporter-user root --exporter-password singlestore
Toolbox is about to set the following monitoring parameters:
· Exporter port: 9104
· Exporter user: “root”
· Exporter password: ●●●●●●

Would you like to continue? [y/N]: y
Operation completed successfully

Let me know if there is a simple solution

Thanks,
Adrian

hniemchenko-ua · August 5, 2021, 9:01am

Hello Adrian, thank you for your post and for trying out SingleStore DB monitoring, glad your issue is resolved.

Your solution is indeed a correct one. SingleStore DB Toolbox, which both sdb-admin and sdb-toolbox-config (or memsql-toolbox-config) are a part of, requires the hosts to be registered in it before performing operations on the nodes located on those hosts. For more information about Toolbox, see SingleStoreDB Cloud · SingleStore Documentation

Hope this explanation helps, feel free to ask any additional questions.