Hi Team,
I have a question about transaction logs. I have set up some S3 buckets as remote storage for certain databases. Then, I created some pipelines to ingest data into a specific database. But when I checked the transaction logs in another database’s partition, I still saw pipeline logs and their access keys. They were hogging most of my bucket space.
Is this normal? And if not, is there a way for me to ensure that only the transaction logs related to a specific database are saved in that database’s designated bucket?
Thank you in advance.
Hi Thao,
Thanks for reaching out to the forums.
If you no longer need the pipelines then those can be dropped.
Try to run the SNAPSHOT commands twice to see if the size is reduced.
I request you to refer to the Understanding Disk Usage for more detailed information on the same.
By using a connection link, you can avoid exposing sensitive information in your SQL statements and transaction logs.
As a best practice if you find that access keys are visible in your logs, take immediate steps to rotate those keys and update your security practices.
Please raise a support request if you are in support of a more detailed analysis of this.
1 Like
Thanks for your response. I tried running the SNAPSHOT command twice and the disk usage was reduced.
But I still wonder if it is normal to record pipeline logs in the database transaction log when that pipeline inserts data into another database?
I think pipeline logs should only be recorded in the transaction log of the database which pipeline inserts data into.