A critical issue has been identified and fixed in SingleStoreDB 7.8, 7.6 and 7.5 that could result in data loss. This issue can occur when using the Unlimited Storage Databases (bottomless) feature installed on premises with AWS S3 storage.
This issue only affects AWS, and those using GCP or Azure bottomless storage are not impacted and require no action.
This issue was caused by a problem with the C++ driver for AWS S3 that allowed a “success” return code to be returned when a blob write actually failed [AWS17, AWS21]. SingleStore has built a version of the AWS driver with a fix, and this version of the driver is now in the latest SingleStoreDB 7.8, 7.6 and 7.5 patch releases. SingleStore is working with AWS to ensure a fix is applied to the original source code repository managed by AWS.
SinglestoreDB Cloud customers running 7.9, 7.8, 7.6 and 7.5 have been or will shortly be upgraded automatically to the latest patches to protect against this issue – no customer action is required.
The symptom of this failure is an error message similar to the following, at the time data is read (queried):
ERROR 2470 (HY000): Leaf Error (<IP address>:3306): Leaf Error (<IP address>:3306): '<db_name>': File </path/to/file> not found in remote.
If your database is corrupted due to this issue, you may repair it, to make it functional again, but possibly with data loss, using
CHECK BLOB CHECKSUM WITH REPAIR FORCE;
Customers using SingleStoreDB with AWS S3 who are not using bottomless are unaffected.
The first patch releases for the major.minor releases mentioned here that have this fix are:
7.9.9
7.8.13
7.6.22
7.5.21
References
[AWS21] Why am I getting a 200 response with “InternalError” or “SlowDown” for copy requests to my Amazon S3 bucket? Resolve Error 200 response when copying objects to Amazon S3 | AWS re:Post, 25 Aug 2021.
[AWS17] S3Client::CompleteMultipartUpload returns success for failed requests, S3Client::CompleteMultipartUpload returns success for failed requests · Issue #658 · aws/aws-sdk-cpp · GitHub, 12 Sept 2017.