Database Restoration Failure: Receiving ERROR 1832 (HY000) Yet Have Available Disk Space

Help
1

Hi,

I’m receiving the following error in mysql when attempting to restore a database pulled from another server, granted I’m attempting to restore from the master aggregator instance of mysql:


ERROR 1832 (HY000): Restoring partitions on db_data. Couldn’t find backup file for partition 0 of database db_data (there are 24 partitions without backup files).


The Operational Errors documentation states “This error means that there is not enough available disk in the backup directory.” However, I have plenty of disk space, roughly 4TB, while the backup is around 100GB. The backup files were copied with the following suffixes, were any missing?:

  • db_data_0.backup
  • db_data_0.backup.columns0.tar
  • db_data_0.backup.columns1.tar
    …
  • db_data_23.backup
  • db_data_23.backup.columns0.tar
  • db_data_23.backup.columns1.tar

https://docs.memsql.com/v7.0/reference/troubleshooting-reference/operations-errors/


Also, this is a 2 server MemSQL Cluster config as documented from the below thread:

2

All files are accesible by the ‘memsql’ account, after performing:

chown memsql:memsql

Still no luck. Here’s the Node view under MemSQL Studio. Those 8TB represent other data, while the database (and backup itself) is under 600GB:

nodes
nodes1199×310 27.3 KB

3

Hello,

What version of memSQL are you running on?

also I saw a small naming bug, the columnar tar files should have the suffix .backup_columns, instead of the .backup.columns

4

Hey nhoran

Ahh yes, I’d manually written those into the forum box, but they do have the proper extension of .backup_colums0.tar & etc.

Source db:
MemSQL Server 7.0.9

Destination db:
MemSQL Server 7.0.15

5

What are you backing up to/restoring from? Cloud storage, network storage, local disk?

It would be helpful if you email your tracelogs of the ma and leaves that are missing partitions to bug-report@memsql.com so I can take a look at them.

6

I am backing up from 1 locally managed cluster (1 master, 1 aggregator, 1 leaf), into another locally managed cluster with the same hardware & number of nodes.

I am not sure how to get those logs.

7

Where did you put the backup files? Are all the partitions on the local disk of the leaf?

8

I have the backup files in the aggregator node directory, on Server 1, while my leaf is on another server.

Should I be restoring it from the same server as the leaf?

9

Yep, so how restore works, the MA will restore the Reference database, in your case named db_data.backup (there also may be columns associated with it), and the leaves will restore the partition databases. So the leaf needs to be able to access all of the partition files {db_data_0.backup, db_data_0.backup_columns*, … }

I hope that helps!

10

Thanks,

I was able to resolve this by copying the backup directory to the <base-install-dir>/data path on both the master aggregator (on Server 1) and the leaf (on Server 2).


From there, I had to change the path and file root permissions on both servers using:

chmod u=rwx,g=rx,o=rx <base-install-dir>/data/<backup-folder-name>/*


Lastly, I had to ensure the 'memsql' user had permissions to the backup path & files, again on both servers: 
chown memsql:memsql <base-install-dir>/data/<backup-folder-name>
chown memsql:memsql <base-install-dir>/data/<backup-folder-name>/*