We’re excited to announce that Stage, SingleStore’s storage platform designed to organize and manage files for seamless ingestion into your SingleStore Helios® database, is now generally available (GA) and included in the Free Tier!
Over the past year, many customers have successfully used Stage during its public preview to streamline their data ingestion processes, providing invaluable feedback that has helped refine this feature. Whether you’re working with small datasets or handling large-scale data, Stage simplifies the process of organizing and preparing your files for ingestion — giving you powerful, enterprise-grade functionality without any added cost.
What is Stage?
As you’ve likely experienced, loading files into a database can be a tedious process. In the past, loading local files into a database required using a local client, manually specifying the schema of the file, and setting up complicated pipeline mappings. This could be time-consuming and prone to errors.
Stage changes that by providing a staging area attached to every starter or standard workspace group, where you can easily upload and organize your data files. Stage simplifies and speeds up the process of loading data into SingleStore. Instead of manually specifying every detail, you can load your files into this staging area and quickly get your data ready for ingestion.
Combined with SingleStore's schema inference, the process of creating tables and setting up pipeline mappings is significantly accelerated. SingleStore automatically detects the structure of your data, making it faster and easier to load data into the database. With Stage, it can take just seconds to go from uploading a file to having it ready for querying.
In addition to importing data, Stage allows you to export query results into the staging area. You can easily share data with other users who have access to the same Stage, or download large result sets to your local machine for further processing. This can be done using the Stage-specific SQL commands we support. Stage supports all file formats including CSV, Parquet, Avro, JSON and compressed formats like .gz, offering flexibility for your data management needs.
Key highlights of Stage
- Simplified file loading. Load files directly into SingleStore using Notebooks or the new Load Data experience.
- Integration with SingleStore's schema inference. Automatically detect the structure of your data, speeding up table creation and pipeline mapping.
- Query result export. Easily export query results into Stage for sharing or local downloads using Stage-specific SQL commands.
- Multiple file formats supported. Stage supports all file formats including CSV, Parquet, Avro, JSON and compressed formats like .gz, offering a versatile solution for data management.
- Built on SingleStore Storage Service. Stage is built on SingleStore Storage Service, a secure, reliable and scalable storage system that powers SingleStore’s most critical features like Notebooks, Secrets, query event history and more.
- Included storage. Stages come with 10GB of storage, accessible across all of your starter or standard workspaces. For additional storage, please reach out to the support team.
Managing files and folders in Stage
Stage provides multiple ways to manage and organize your files and folders before loading them into your database. This flexibility allows you to choose the best method for your workflow:
Cloud Portal UI
Our Cloud Portal offers a user-friendly interface to upload, view and organize your files. This graphical interface is ideal for users who prefer an intuitive, visual way to manage their data.
Notebooks and SQL commands
With SingleStore Notebooks, you can manage your staged files using SQL queries or FUSION SQL directly within the notebook environment. This allows you to easily load and interact with your data in Stage for easy analysis and processing, all in one place. Here are some examples of SQL commands you can use with Stage:
Exporting query results. You can easily export your query results into Stage. Note: To avoid having multiple files from each leaf node, we are using “GROUP BY 1”.
SELECT * FROM <table_name> GROUP BY 1 INTO STAGE '<table_results.csv>';
CREATE PIPELINE <pipeline_name>AS LOAD DATA STAGE <path_in_Stage/filename> { <pipeline_options> }INTO TABLE <table_name>{ <data_format_options> }
SingleStore Python Client
If you’re working with Python, you can use the SingleStore Python Client to manage your files and folders in Stage programmatically. This allows for seamless integration into Python-based data workflows, from organizing files to loading them into your SingleStore Helios databases.
Management API
For developers looking to automate file management or integrate file organization into existing workflows, the Management API provides powerful control. You can upload and manage files and directories via programmatic HTTP requests, enabling seamless automation.
For more details, please refer to the Stage section in our Management API Reference. These management options give you flexibility and control over your data, whether you prefer a graphical interface, API-based automation or working directly in a notebook or Python environment.
For further guidance on how to use Stage, check out our documentation or connect with our support team!
Get started with Stage on Free Tier today!
Now that Stage is generally available and part of the Free Tier, there’s no better time to start organizing and managing your data more effectively. Sign up for SingleStore Free Tier today and experience how Stage can simplify and streamline the process of preparing your data for ingestion into SingleStore Helios databases.
