Best practices to prevent Disk-full?

Help
1

Hi. everyone!!

The leaf node of our cluster went OFFLINE because the disk was full.

In our cluster, only a few SQL (SELECT, INSERT, DELTE) occur and a lot of data is INSERTed in the columnstore table.

We want to prevent it from going OFFLINE because the disk is full.

Do you have any other options than the following two?
1. Periodic external monitoring of disk state
2. View information_schema.MV_SYSINFO_DISK, LMV_SYSINFO_DISK

We want to hear various opinions.

Thank you in advance.

2

Hello,

Thank you for reaching out.

The two options you mentioned are good solutions, a couple of other options you can consider:

  1. In SingleStore Studio (our management tool) version 2.0.3, we added the disk usage compared the total utilization on the main dashboard, which can give you real-time information on your disk usage compared to how much you have available.
  2. If you use SingleStore DB Toolbox to deploy and manage your cluster, you can use our report module. The report module allows you to collect a diagnostic report and check it against various best practice recommendations. The report does a one-time check, but you can schedule it with a tool like cron. One of the checkers is called diskUsage and reports on if you are close to hitting limits. If the tool identifies disk pressure, it will provide a WARN or FAIL output in the check output depending on how close you are to the limit. We just recently (this week) introduced a command in this module called send-alert. This allows you to provide an SMTP configuration to the report check so that you can email yourself the output of this report and identify issues faster. I’ve linked the command information and we will also be publishing a guide shortly, so please stay tuned.
  3. Related to Periodic External monitoring, we also have an official Monitoring solution that you can configure here - this comes with pre-built Grafana dashboards that monitor disk: SingleStoreDB Cloud · SingleStore Documentation. Note that you need to be on the engine version of at least 7.1.8 to set up this solution.

Hope this helps.

Cheers,
Roxanna

1 Like
3

Hi @roxannapourzand,
That’s great new.
We use Datadog at Hulu. Just wondering if there are an plans to provide datadog templates for MemSQL?

4

Hi Franck,

We don’t have any near terms plans to provide DataDog templates. That said, this feedback is useful, and a few other customers have also reported using DataDog; I’ve added you as a requester for this feature.

Would you mind also going to the feature requests page in our forum and filing it there? This will also allow other members of our community to vote on your request.

Best,
Roxanna