(Ported from memsql-public-chat Slack channel)
Matteo [4:18 AM]
hi all, I’m totally new of memsql and I’m testing it for a pilot project in our company
unfortunately I didn’t notice I was filling the disk with data and once noticed it was too late… now, despite the fact there’s room enough to grow, anytime I try to perform even just a select on my database, I receive the following erro:
ERROR 1777 (HY000): Partition aic_monitor:0 has no master instance.
rebooting even the whole server didn’t resolve… and both master and leaf (which are on the same machine) are alive
is there any chance I can recover my data or I’d better restart from scratch? of course that thing worries me if I have to move to production…
seth [10:01 AM]
@Matteo it sounds like you are missing master partitions for the database… which isn’t good. master partitions are the active partitions for your database.
select @@redundancy_level;
select @@default_partitions_per_leaf;
show aggregators;
show leaves;
show cluster status;
use database_name;
show partitions extended;
^^ run these commands on your master aggregator and share the output.
Matteo [11:35 AM]
thank you @seth! going to run them all as soon as I reach the dev server!
Matteo [2:10 AM]
I’ve just anonymized the IP address, that’s the status of the db
mysql> select @@redundancy_level;
+--------------------+
| @@redundancy_level |
+--------------------+
| 1 |
+--------------------+
1 row in set (0.12 sec)
mysql> select @@default_partitions_per_leaf;
+-------------------------------+
| @@default_partitions_per_leaf |
+-------------------------------+
| 8 |
+-------------------------------+
1 row in set (0.13 sec)
mysql> show aggregators;
+----------------+------+--------+--------------------+------------------------------+-------------------+--------+
| Host | Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | Master_Aggregator | NodeId |
+----------------+------+--------+--------------------+------------------------------+-------------------+--------+
| <VALID IP> | 3306 | online | 1 | NULL | 1 | 1 |
+----------------+------+--------+--------------------+------------------------------+-------------------+--------+
1 row in set (0.00 sec)
mysql> show leaves;
+----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+--------+
| Host | Port | Availability_Group | Pair_Host | Pair_Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | NodeId |
+----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+--------+
| <VALID IP> | 3307 | 1 | NULL | NULL | online | 10 | 0.349 | 2 |
+----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+--------+
1 row in set (0.00 sec)
mysql> show cluster status;
+---------+----------------+------+---------------+-------------+-------------+---------------+----------------+-------------+-------------------------+----------------------+----------------------+-----------------+-------------------------------------------------+
| Node ID | Host | Port | Database | Role | State | Position | Master Host | Master Port | Metadata Master Node ID | Metadata Master Host | Metadata Master Port | Metadata Role | Details |
+---------+----------------+------+---------------+-------------+-------------+---------------+----------------+-------------+-------------------------+----------------------+----------------------+-----------------+-------------------------------------------------+
| 1 | <VALID IP> | 3306 | aic_monitor | master | online | 30:10 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 1 | <VALID IP> | 3306 | aic_stats | master | online | 0:11650 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 1 | <VALID IP> | 3306 | cluster | master | online | 5:18 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 2 | <VALID IP> | 3307 | aic_monitor | sync slave | replicating | 30:10 | <VALID IP> | 3306 | 1 | <VALID IP> | 3306 | Reference | |
| 2 | <VALID IP> | 3307 | aic_monitor_0 | detached | pending | 439:265207550 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_1 | detached | pending | 439:262971139 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_2 | detached | pending | 439:261157609 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_3 | detached | pending | 439:267582197 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_4 | detached | pending | 439:260992115 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_5 | detached | pending | 439:259062765 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_6 | detached | pending | 440:10 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_monitor_7 | detached | pending | 439:267115988 | NULL | NULL | NULL | NULL | NULL | Detached Master | |
| 2 | <VALID IP> | 3307 | aic_stats | sync slave | replicating | 0:11650 | <VALID IP> | 3306 | 1 | <VALID IP> | 3306 | Reference | |
| 2 | <VALID IP> | 3307 | aic_stats_0 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_1 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_2 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_3 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_4 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_5 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_6 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | aic_stats_7 | master | online | 0:11393 | NULL | NULL | NULL | NULL | NULL | Master | |
| 2 | <VALID IP> | 3307 | cluster | async slave | replicating | 5:17 | <VALID IP> | 3306 | 1 | <VALID IP> | 3306 | Reference | stage: packet wait, state: x_streaming, err: no |
+---------+----------------+------+---------------+-------------+-------------+---------------+----------------+-------------+-------------------------+----------------------+----------------------+-----------------+-------------------------------------------------+
22 rows in set (0.01 sec)
mysql> show partitions extended;
+---------+------+------+------+--------+------+--------------+------------+--------------+
| Ordinal | Host | Port | Role | Locked | Info | Last Command | Last Error | Last Message |
+---------+------+------+------+--------+------+--------------+------------+--------------+
| 0 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 1 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 2 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 3 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 4 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 5 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 6 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
| 7 | NULL | NULL | NULL | 0 | NULL | NULL | NULL | NULL |
+---------+------+------+------+--------+------+--------------+------------+--------------+
8 rows in set (0.00 sec)
Matteo [3:33 AM]
@seth thank you for your hints: I’ve checked the “detached” leaves, I’ve re-detached them and attached again and it did the trick! I can now see data again!