Hi SingleStore,
I’m trying to restore a 2GB backup from GCS on a fresh cluster-in-a-box (7.5) locally (docker with 6 CPU + 10GB RAM).
After few minutes of processing the query crashes with:
ERROR 1970 (HY000): Leaf Error (127.0.0.1:3307): Remote connection timed out waiting for a response from the external endpoint. No stderr returned.
It’s impossible for me to dev locally anymore.
How to fix this please?
Thanks
Hi Pierre,
That error is a result of very slow download from GCS (I believe the default timeout is subprocess_io_idle_timeout_ms before giving that error). Is the failure consistent ?
We’re likely going to need a cluster report to check on things in a bit more detail.
After inscreasing the subprocess_io_idle_timeout_ms to 10 minutes, I still have the same error, and the first error in the logs is:
ERROR: Thread 99995: createDatabaseHelper: my_db_name: db creation fails because 'HandleBackupTargetForCreateDb' failed with error 5
Also, I tried to download manually the GCS backup, and restore it with RESTORE DATABASE xxx FROM ‘/path’ but it fails to construct to proper path to locate the backup file.
How to RESTORE an S3/GCS backup from a local path please?
Hi Pierre,
Local backups unfortunately have a different format than blob store backups, so directly restoring a S3/GCS backup from a local path will not work. However, it is possible to hack a S3/GCS backup to be able to be restored locally. If you append “.backup” to all the files except for the manifest files and BACKUP_COMPLETE, and restore from /path/[dbname].backup, the restore should work. For example, change db1_0 to db1_0.backup …etc.
As for GCS failing to restore, it is not super clear at the moment. Could you try setting subprocess_io_idle_timeout to 30 minutes as a sanity test and if that fails generate a cluster report?
Thanks @jhuang for the trick, with subprocess_io_idle_timeout to 30 mins, the DB RESTORE worked, tried twice and it completed.
The RESTORE query took a bit more than 10 mins… as long as it works, for a dev env it’s fine!