[Spce-user] replicate spce servers

Tue Jul 18 18:53:19 EDT 2017

Hi All,

We use a fairly simple backup/restore process here to ensure availability. It should be noted that in a failover, we don’t care about call records; we bill externally by channel. All that matters is that calls go through while we restore the primary host. Call data we can recover externally, if required. In this setup, we can be down for around 15 mins while we fail over. Usually less. We keep the DNS entries TTL super low. It’s not ideal, but it’s not being completely down all day either. Most customers understand small outages, as long as they are recovered quickly. We’ve had to failover once years ago, we lost connectivity at our host and it was out of our control. Otherwise, we’ve been rock solid.

Our primary box is segregated and runs on physical hardware. The backup is on ESXi within our corporate infrastructure. This is also where it gets access to cold backup storage. We don’t do any fancy IP stuff. Keeping them as separate systems that sync config once per day. In the event of a failover, we simply update the DNS records of the primary domain. Since some of our carriers won’t failover automatically anyway, we have to move them manually. We have both systems IP’s setup as domains, purely to handle incoming calls via our peers.

We previously tried MySQL replication with floating IPs and what not but just didn’t need the complexity of always replicating systems. Since we really only need subscriber info to sync, once a day is fine for us. If we go down, worst case is we have to manually add a customer or 2 if they were not in the backup from the night before.

If the primary fails, we move to new hardware and begin restoring the primary box immediately, moving customers back as soon as possible. It has never been necessary.

Linked is the script I was using on our mr3 infrastructure prior to migrating to mr5 on new hardware. I was hoping to use the inbuilt backup system instead but it don’t seem to work. So we’ll be moving back to something similar again real soon. It seems like it should work fine on mr5 but I have not tested it in production with that version.

https://gist.github.com/maxwellpower/b087e19b73dd13aeed4c9f825fc5b3c1

Cheers,
Maxwell Power

Technical Account Manager
Phone: (780) 809-9990 Ext. 417 | Toll Free: (855) 4-YEGTEL(934835) | Fax: (780) 401-3390
YEGTEL Communications INC. | 10301 104 ST NW, Suite 55, Edmonton, Alberta, T5J 1B9
Email:  <mailto:mpower at yegtel.ca> mpower at yegtel.ca | Web:  <https://www.yegtel.ca/> www.yegtel.ca

This e-mail and any attachments may contain confidential or privileged information. If you are not an intended recipient, do not re-send, copy or use this e-mail. Please also contact the sender immediately and delete this e-mail in its entirety. Privilege is not waived by reason of mistaken delivery to you. YEGTEL Communications and its affiliates accept no liability whatsoever for loss or damage in relation to this e-mail and may monitor, retain and/or review email. Opinions expressed in this e-mail are those of the author and may not represent the opinions of YEGTEL Communications and its affiliates.

Ce courriel et toutes ses pièces jointes peuvent contenir de l'information de nature confidentielle ou privilégiée. Si vous avez reçu ce courriel par erreur, merci de ne pas le transférer, le copier ou l'utiliser. Veuillez communiquer immédiatement avec l'expéditeur et supprimer le message dans son intégralité. Le fait de vous avoir envoyé ce courriel par erreur ne signifie pas que l'expéditeur renonce à ses droits. YEGTEL Communications et ses sociétés affiliées ne peuvent être tenues responsables de toute perte ou dommages liés au présent courriel et peuvent effectuer un suivi de ce courriel, le conserver et l'examiner. Les opinions exprimées dans le présent courriel sont celles de son auteur et non celles de YEGTEL Communications et de ses sociétés affiliées.

From: Spce-user [mailto:spce-user-bounces at lists.sipwise.com] On Behalf Of Stephen Donovan
Sent: July 18, 2017 15:25
To: William Fulton <wfulton at thirdhatch.com>
Cc: Spce-user <spce-user at lists.sipwise.com>
Subject: Re: [Spce-user] replicate spce servers

I can't speak for Alex, but we are using xenserver 7 on dell hardware, storage is iscsi on NAS4free, 15krpm SAS drives or SSD.  ZFS RAIDZ2.

Stephen

  _____  

From: "William Fulton" < <mailto:wfulton at thirdhatch.com> wfulton at thirdhatch.com>
To: "Alex Pearson" < <mailto:apearson at questblue.com> apearson at questblue.com>, "Dan Boghici" < <mailto:dboghici at sipwise.com> dboghici at sipwise.com>, "Spce-user" < <mailto:spce-user at lists.sipwise.com> spce-user at lists.sipwise.com>
Sent: Tuesday, July 18, 2017 12:56:57 PM
Subject: Re: [Spce-user] replicate spce servers

Alex,

I am curious about your performance in virtualization. We have tested it on an Amazon image and had mixed result in regards to performance. So have only used bare metal in production. What kind of virtualization are you using and how is your performance?

Thanks,

Bill

From: Spce-user [ <mailto:spce-user-bounces at lists.sipwise.com> mailto:spce-user-bounces at lists.sipwise.com] On Behalf Of Alex Pearson
Sent: Tuesday, July 18, 2017 10:47 AM
To: Dan Boghici < <mailto:dboghici at sipwise.com> dboghici at sipwise.com>;  <mailto:spce-user at lists.sipwise.com> spce-user at lists.sipwise.com
Subject: Re: [Spce-user] replicate spce servers

Good Morning,

Regarding the RTO, Great Question!  The majority the discussions with our team on this project have been focused on recovery.  

The RTO for the proposed solution is essentially 0 as there is no disruption of operation.  

I’ll build upon my previous response by adding some of the key items we discussed that led to the current configuration.  

After reviewing our current infrastructure we determined that we could run more efficiently by migrating to a virtual environment.

We used two host machines to split the infrastructure into two separate points of failure.  Sipwise A and MYSQL A on Host A.  Sipwise B and MYSQL B on Host B.

We keep them highly available using keepalived.  The Sipwise servers share a floating IP and the MYSQL servers share a floating IP.

If a host goes down, the images will automatically failover to the images on the other Host.

Once that was completed all images were shut down, copied, and stored both on the local machine and a remote machine.  

We now have a clone of our entire Sipwise environment that can be moved and powered on whenever or wherever we see fit.  

If a Sipwise or MYSQL image goes down you can replace it with a back up image with identical configuration. Which is why we wanted our data on a separate image.

In the event of a complete disaster, you maintain control of the images.  If you properly backup your data, you could dump a data backup onto the cloned system and be up and running.

I hope this helps answer the question.  Also, I am open to suggestions/ideas on ways to improve.

Regards,

Alex Pearson

From: Spce-user [ <mailto:spce-user-bounces at lists.sipwise.com> mailto:spce-user-bounces at lists.sipwise.com] On Behalf Of Dan Boghici
Sent: Tuesday, July 18, 2017 03:10
To:  <mailto:spce-user at lists.sipwise.com> spce-user at lists.sipwise.com
Subject: Re: [Spce-user] replicate spce servers

why not buy the PRO version and save yourself unnecessary headache?

Dan

On 07/18/2017 12:07 AM, Alex Pearson wrote:

Good Afternoon,

Yes you can.  For example, we have a Sipwise A and a Sipwise B configured with a floating IP.  Then we have a MYSQL A server and MYSQL B server configured with Master-Master  replication and a floating IP.    The Sipwise A and Sipwise B servers are configured to use the floating IP of the database servers.

If Sipwise A dies, Sipwise B automatically takes over.  If MYSQL A goes down, MYSQL B takes over.  Effectively, ensuring that you are able to route calls AND secure your data in the event of a failover situation.  I hope this helps!

Alex

From: Spce-user [ <mailto:spce-user-bounces at lists.sipwise.com> mailto:spce-user-bounces at lists.sipwise.com] On Behalf Of Stephen Donovan
Sent: Monday, July 10, 2017 19:26
To: Spce-user
Subject: [Spce-user] replicate spce servers

Hello,

Say I wanted to replicate the databases of two spce instances where they have different ip addresses but all else is the same, same domain, etc.  the idea is to allow customers to connect to either server and process calls.  Inbound calling, I can handle the failover externally.  2nd instance would be for backup only in a remote datacenter.  I would use DNS SRV records and compatible endpoints to manage the failover for the same hostname.  I'm basically asking can I replicate the databases and expect this to work if needed in a failure scenario?

Stephen

_______________________________________________
Spce-user mailing list
Spce-user at lists.sipwise.com <mailto:Spce-user at lists.sipwise.com> 
https://lists.sipwise.com/listinfo/spce-user

_______________________________________________
Spce-user mailing list
 <mailto:Spce-user at lists.sipwise.com> Spce-user at lists.sipwise.com
 <https://lists.sipwise.com/listinfo/spce-user> https://lists.sipwise.com/listinfo/spce-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20170718/dc6939bd/attachment-0001.html>