SingleStore Storage Service is the backbone of many of our key features, including Notebooks and Stage. Built on top of cloud blob storage, it adds a layer of abstraction and additional features tailored to our specific use cases, making it the perfect solution for our storage needs.
We have been using MinIO to mock Amazon S3 for AWS in our development environment. As part of our efforts to improve our processes, we recently decided to move away from MinIO and start using LocalStack. In this blog post, we will present the reasons behind our decision — and what we learned from it.
What is MinIO?
MinIO has been a crucial tool in our development workflow, allowing us to mock Amazon S3 for local development and testing. With MinIO, we can simulate the behavior of an Amazon S3-compatible storage service, enabling us to develop and test our applications without relying on a live environment. Although this has proven invaluable for speeding up our development cycles and ensuring the reliability of our features, it has also presented some challenges.
While MinIO offers a robust set of features for object storage, it does not fully replicate the complete range of capabilities offered by Amazon S3. This limitation impacted our ability to fully simulate the environment for testing and development purposes, as certain features were missing or behaved differently. As a result, our test coverage was reduced since we were forced to skip some integration tests in the local development.
An example of this is how the naming differs between both. While in Amazon S3 a file and a directory can have the same name to allow for a hierarchical structure, this behavior is not supported in MinIO, as it follows a strict object storage model where each object must have a unique name (regardless of its type). Supporting these discrepancies would add extra complexity to our development process.
Another significant challenge we faced was the need to maintain two sets of codebases — one for local development using the MinIO SDK and another for production using the AWS SDK. This dual codebase approach introduced complexity and overhead, as any changes or updates to the production codebase had to be manually synchronized with the MinIO SDK-based codebase used for local development.
This process was not only time consuming but prone to errors, since discrepancies between the two codebases could lead to inconsistencies and issues in our applications. To avoid them, we were thoroughly testing our changes in the staging environment to use the live cloud blob storage environment. As expected, this added an extra layer of complexity and time consumption during our development process. After some investigation, we decided to migrate to LocalStack.
You can find a detailed comparison here.
What is LocalStack?
The goal of LocalStack is similar to MinIO: emulate cloud services that allow developers to run them locally on their machines and in their CI environment. It's designed to facilitate the development, testing and debugging of applications that use AWS.
From our experience, using LocalStack over MinIO offered several advantages — we’ll explore them here next.
LocalStack addresses the discrepancies and limitations encountered with MinIO by providing a more comprehensive emulation of the AWS environment. While MinIO is excellent for mocking S3, LocalStack's ability to emulate the entire AWS environment ensures that our local development environment closely mirrors the production AWS environment.
This capability allows us to create more realistic testing scenarios, which increases our ability to catch potential issues early in the development cycle. This led to an improvement in the overall stability and reliability of our services.
We also took advantage of LocalStack's seamless integration with the AWS SDK, eliminating the need to maintain separate codebases for local development and production environments. We streamlined our development process and reduced complexity using the AWS SDK for both LocalStack and production. This change allowed us to remove approximately 2,000 lines of code that were solely responsible for running MinIO.
Challenges faced in our migration
One of the biggest challenges with LocalStack is data persistence. Version v1 of LocalStack supported data persistence, allowing us to mount the volumes and set the LocalStack data directory using the DATA_DIR environment value. However, in the latest release of LocalStack, (>= v2.0.0) the persistence feature is removed from the community version and is only made available in the pro version — so we need to be extra careful not to overload the CPU or memory.
With this migration, we modified our test suites and testing workflow by deploying LocalStack in a Kubernetes cluster for our testing environment. To streamline our testing process, we need a mechanism to automatically provision S3 buckets as soon as the LocalStack service is up and running. To achieve this, we configured LocalStack to run a script using its startup script feature, ensuring the script runs as soon as the LocalStack service is ready.
Closing thoughts
The migration from MinIO to LocalStack has been a significant step forward for our team. Not only has it addressed the limitations and discrepancies we encountered with MinIO, but it has also helped us reduce complexity and remove code. We are now able to create a testing environment that closely mirrors our production environment, making us more confident about what we are shipping — and the reliability and functionality of our services.
We are excited to leverage these possibilities in future projects, further enhancing our development and testing capabilities!
