Crazy disk usage in data/rem

For some reason we are collecting massive amounts of data in data/rem with files named like:

metrics_singlestore-leaf-02:3306_963371508_50400794.rem

We run SingleStore DB Version: 8.9.2, it started after the upgrade (we were on 8.7 beforehand)

We have 4.7GB in our database but somehow 91GB in this data/rem folder

How can I clear it out? I have tried googling for ages but can’t find any mention of it anywhere, e.g its not on https://support.singlestore.com/hc/en-us/articles/4406838603156-Understanding-Disk-Usage at all

Now updated to 8.9.4 after spotting this in the release notes and hoping its related?

  • Bugfix: Resolved an issue where an in-development subsystem can leak files on disk

Will I need to delete these files manually or does SingleStore clean these up for me?

Yep, that sorted it!

So if anyone is wondering if they’re in the same position, update your software :joy:

Would be nice if singlestore documented this somewhere as nothing anywhere mentions these files, even the line in the changelog for the bug fix just hints at some mysterious files existing but not what they are or where they are

I’m having to clean them up manually, it would be nice if SingleStore took care of it though