Cross data center Cluster Failover

1

HI Team,

Need help…!!

Env. two singlestore clusters in two different DC.
Challenge: TO configure HA between these two clusters…

  1. We have enabled asyn replication for the databases between these clusters and cron monitors the health on both sides if primary goes down, job will stop the replication between clusters. This is sorted.
  2. Now at application level how do we configure the auto fail-over for connections coming form outside, if the primary cluster goes how do we auto switch to secondary…?
2

Configure health checks in your load balancer to regularly check the availability of the primary cluster. These health checks should test the SingleStore cluster’s health, such as by checking its responsiveness or the status of replication. If the primary cluster becomes unresponsive, the load balancer should automatically direct traffic to the secondary cluster. Alternatively, you can use a DNS failover service. These services automatically update DNS records to point to the IP address of the secondary cluster when a failure is detected in the primary cluster. Popular DNS failover services include Amazon Route 53, Cloudflare Load Balancing, and Dyn.

1 Like
3

Thank you for the response.

We have created the two API’s to monitor the health check of the cluster and return the status “Good/bad”. Based on the API status we have configured the failover. Good No action required, Bad Failover.
At n/w level we use F5 for load balancing, the challenge is it can only monitor the Primary API and if the primary API goes down connections will be routed to secondary and we stop replication using the crons. But once the primary is back up the connections are routed back again to primary which creates a situation were we have different data on the clusters, as the singlestore replication is Asyn.

Any suggestions,?? We can only use f5 for load balancing…