I did PITR(Point-in-Time Recovery) performance test with Bottomless Database (in v.7.5.beta).
There was a problem detaching the database after checking its normal behavior.
I should have deleted the data files in S3 after I ran the DETACH command in SingleStore.
However, that command does not work in spite of using the force option, because I deleted the files in S3, first.
I deleted all the tables in the DB, stopped and restarted the entire node.
But, the Bottomless Database is now unable to command use.
Furthermore, some commands such as CREATE DATABASE and TRUNCATE were executed slowly or failed.
The current state of the cluster is as follows.
$ sdb-admin list-nodes
±-----------±-------±--------------±------±--------------±-------------±--------±---------------±-------------------±-------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
±-----------±-------±--------------±------±--------------±-------------±--------±---------------±-------------------±-------------+
| 4E63264C4B | Master | 192.168.1.200 | 30130 | Running | True | 7.5.2 | RecoveryFailed | | 0.0.0.0 |
| 4D4A3D4152 | Leaf | 192.168.1.200 | 30131 | Running | True | 7.5.2 | RecoveryFailed | 1 | 0.0.0.0 |
| 58065EFDD6 | Leaf | 192.168.1.200 | 30132 | Running | True | 7.5.2 | RecoveryFailed | 2 | 0.0.0.0 |
±-----------±-------±--------------±------±--------------±-------------±--------±---------------±-------------------±-------------+
singlestore> show databases;
+--------------------+
| Database |
+--------------------+
| cluster |
| information_schema |
| memsql |
| metrics | > Local Database
| mk_bottom | > Bottomless Database
+--------------------+
singlestore> use mk_bottom;
ERROR 1768 (HY000): The database 'mk_bottom' is corrupted and can't be recovered. Please contact technical support.
singlestore> CREATE DATABASE IF NOT EXISTS test;
Query OK, 1 row affected, 6 warnings (**8 min 1.15 sec**)
singlestore> create table t1 (a int, b int);
Query OK, 0 rows affected (0.02 sec)
singlestore> insert into t1 values(1,2);
Query OK, 1 row affected (0.03 sec)
singlestore> truncate t1;
ERROR 2282 (HY000): Failed to synchronize database. Timed out waiting for the ReplicationManagement thread to synchronize
How can I forced detach the bottomless database in this case?
Is the reason for the slow CREATE DATABASE command related to Bottomless Database?
And How can I solved that?
Can we see the error that DETACH … FORCE is giving you? That is the command that should drop the database no matter the state of S3. I’ve opened an internal investigation to try and reproduce this ourselves in the meantime.