Leaf node goes offline during operation

Help Connectors and Pipelines
1

2hosts (MA, 1LEAF) is configured without HA.
Leafnode went offline while fetching 270,000 files from the pipeline.
I need to know why it went offline.
What is the solution for this?
Please refer to the information below.

It is also not a repair database.
delete the DB and try again?

image
image.png815×182 4.36 KB

image
image.png1708×408 12.9 KB

image
image.png1103×123 3.93 KB

image

thanks.

2

Hello chaeyoung.ko,

Sorry, for the late response.
Were you trying to load data into a RowStore table or a ColumnStore table? What is the size of the total data? How much memory do you have on the leaf node?
When loading into RowStore table via Pipelines there is a chance of you running out of memory.
Please look into the above and get back to us with the details.

Thanks,
Ramesh

3

We made up a target table with a column store.

This happened while importing data from the application server, which generates a file approximately 44 mb in size every second, to the db server via a file system pipeline.

I set max_memory to 128G. So, the leaf node has memory of 128G.

thanks.

4

Hello chaeyoung.ko

Thanks for the information. Here are a few observations based on the information you provided.

  1. You are using ColumnStore table which is recommended for high volume pipeline tasks.
  2. The leaf stops after about 270,000 files each 44 MB. Based on the calculation, it would have stopped after 75 hours and after inserting almost 11.3 TB of data. Is that an accurate estimate?
  3. Have you set the max_memory to 128GB. How much is the total host memory? How much disk space do you have on the host machine? Please see this document link for memory limits, what can happen and what you can do.
    SingleStoreDB Cloud · SingleStore Documentation
  4. What are the error messages you see in the MemSQL logs?

Hope this helps to investigate further.

Thanks,
Ramesh