we are getting more and more into pipelines and are pretty amazed by the performance!
In order to ease our development and management, I would like to propose improvements to Studio:
GUI to create / update Pipelines (e.g. to select the data source, to select the table, to manage the transforms)
When modifying a pipeline, it is important to start it again correctly so that the previously imported files are not imported again. It is easy to make mistakes here, so a GUI would help
Some statistics relating to the amount of imported data and the speed of which this happens
I lead the MemSQL Studio development at MemSQL and I agree with you that we need a GUI to create pipelines. We’re still in the very early stages of investigating how we’d build this, and so we appreciate your bullet point list of things you’d like see.
I think we can keep this thread open in case anybody else wants to give us more feedback on things they’d like to see here (and why).
Shouldn’t be too difficult to build an interface in Studio. Theres only a handful of options. The biggest hurdle would be being able to pull a file to hold in memory to allow mapping (if using the JSON format)