Version 7.3.2 cluster-in-a-box leaf node becomes irresponsible and it starts recovery process. It happens approximately two-hourly. Config is Redhat 7/128Mbyte/32 thread. According to the logs we cannot find any useful information to move forward. Is there another logs that we can analyze or any way to get more “debugging” data? Thanks in advance for any guidance.
==> /var/lib/memsql/22b68876-220e-4d42-89dd-79dff060836d/tracelogs/memsql.log <==
9480253752 2021-03-29 12:32:45.984 ERROR: ProcessNetworkEvents Heartbeat connection error receiving server handshake packet while connecting to 127.0.0.1:3307 (104 = Connection reset by peer)
9480253974 2021-03-29 12:32:45.984 ERROR: db_management log: Disconnecting slave at node 2 because of connection error.
9480254300 2021-03-29 12:32:45.985 ERROR: ProcessNetworkEvents Failure during heartbeat connecting to 127.0.0.1:3307
9480254858 2021-03-29 12:32:45.985 ERROR: db log: Disconnecting slave at node 2 because of connection error.
9480259121 2021-03-29 12:32:45.985 ERROR: cluster log: Disconnecting slave at node 2 because of connection error.
9480259145 2021-03-29 12:32:45.987 WARN: Fail to create connection during clear namespace. Skipping leaf ‘127.0.0.1’:3307.
9480259363 2021-03-29 12:32:45.990 INFO: Thread 115089: DisconnectSlave: cluster log: Removed slave at node 2 because of disconnect.
9480259387 2021-03-29 12:32:45.990 INFO: Thread 115081: DisconnectSlave: db_management log: Removed slave at node 2 because of disconnect.
9480259413 2021-03-29 12:32:45.990 INFO: Thread 115087: DisconnectSlave: db log: Removed slave at node 2 because of disconnect.
==> /var/lib/memsql/22b68876-220e-4d42-89dd-79dff060836d/tracelogs/memsql.log <==
9480776656 2021-03-29 12:32:46.507 ERROR: ProcessNetworkEvents Heartbeat connection error connecting to 127.0.0.1:3307 (111 = Connection refused)
9480776703 2021-03-29 12:32:46.507 ERROR: ProcessNetworkEvents Failure during heartbeat connecting to 127.0.0.1:3307
9480809097 2021-03-29 12:32:46.539 INFO: ProcessTransactions Thread 114960: HandleLeafFailure: Leaf 127.0.0.1:3307 is no longer reachable and will transition from ‘online’ to ‘offline’
9480811562 2021-03-29 12:32:46.542 INFO: ProcessTransactions Flushed 0 connections to ‘127.0.0.1’:3307
9480811596 2021-03-29 12:32:46.542 INFO: ProcessTransactions Thread 114960: SaveState: Leaf 2 (127.0.0.1:3307) has transitioned from ‘online’ to ‘offline’
9480811619 2021-03-29 12:32:46.542 INFO: ProcessTransactions Node 127.0.0.1:3307 heartbeat failure summary. Initial heartbeat failure at 2021-03-29 12:32:27. 206 Consecutively Missed heartbeats. Failover was triggered after 200 missed heartbeats
9480811637 2021-03-29 12:32:46.542 INFO: ProcessTransactions Heartbeat connection attempts summary:
9480811655 2021-03-29 12:32:46.542 INFO: ProcessTransactions Node 127.0.0.1:3307 encountered a heartbeat connection failure (104:Connection reset by peer) at 2021-03-29 12:32:45
9480811675 2021-03-29 12:32:46.542 INFO: ProcessTransactions Node 127.0.0.1:3307 encountered a heartbeat connection failure (111:Connection refused) at 2021-03-29 12:32:46
9480811689 2021-03-29 12:32:46.542 INFO: ProcessTransactions Node 127.0.0.1:3307 heartbeat is currently attempting to reconnect
9480951225 2021-03-29 12:32:46.681 INFO: Thread 114955: FailoverLeaf: Failing over from node 2.