Subject: [codesprints] Failover report

From: Glen Barney <[email-masked]>
Date: Mon, 5 Sep 2011 17:59:50 -0700

Dear IETF Leadership -

At 3:14 PM Pacific Time today, Monday, September 5th, we received an alert
from our monitoring systems that power at our primary facility had failed.
Of course, we have battery backup here, and were still in operation.

I responded to the facility, arriving approximately 20 minutes later, and
confirmed that power was out.  This was not difficult, as the traffic signals
in the entire area were out as well.

PG&E crews were arriving just as I did, and stated that they expected a
restoration time no earlier than 5:30 PM.

As a result, I executed our failover procedure, and transferred IETF operations
to the Orlando data center.

As with any such situation, some users may have encountered interruptions in
service.  However, our failover was completed very quickly, so any problems
that were experienced were minimal.

I will be remaining here for a bit longer until the PG&E crews leave, just to
make sure that another power outage isn't happening.

At 8PM Pacific time, I will initiate a planned outage, to do a controlled
fail-back to Fremont.  I will announce this planned outage on the IETF list
momentarily.

As always, if there are any questions, please feel free to let me know.

Thanks,
Glen

---

From: Russ Housley <[email-masked]>
Date: Mon, 5 Sep 2011 21:06:44 -0400

Given this situation, it sounds like you made the right call.  You might want to leave things running in Orlando until tomorrow.  Your call ...

Russ

On Sep 5, 2011, at 8:59 PM, Glen Barney wrote:

> _______________________________________________
> IAOC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/iaoc

---

From: Glen <[email-masked]>
Date: Mon, 5 Sep 2011 18:40:09 -0700

Dear IETF Community -

Earlier today a local-area power outage of several square miles occurred in
the area of our Fremont data center.  Since that left us running on backup,
we executed an emergency failover to our Orlando data center, where we are
currently running.

Power has been restored, and I will be verifying full completion with the 
power crews shortly.  Barring additional issues, I plan to transfer control
back to our Fremont data center at 8:30 PM Pacific Time (4:30 AM GMT) this o
evening.

During the transfer, web and email services will be interrupted.  We expect
the interruption will take approximately 20 minutes.  There is no need to 
panic, I will be handling the failback personally, and will have service
restored as soon as possible.

Thank you for your patience.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Glen <[email-masked]>
Date: Mon, 5 Sep 2011 20:00:35 -0700

On Mon, Sep 05, 2011 at 09:44:52PM -0500, Pete Resnick wrote:
> Hi Glen,
> The currently running datatracker has older data than the one I was using 
> earlier today. Is that is to be expected in an emergency failover?
> Just checking.

Certainly not.  The MySQL database is synced on a transactional basis.  
Every transaction committed to the database is immediately committed on
all three servers.  And the process was running correctly the entire time.

In the interest of haste, I am contacting you on IM right now, as you already
know.

Glen

---

From: Glen <[email-masked]>
Date: Mon, 5 Sep 2011 20:02:46 -0700

> earlier today. Is that is to be expected in an emergency failover?

I have the information from Pete and am checking now.

Glen

---

From: Glen <[email-masked]>
Date: Mon, 5 Sep 2011 20:19:50 -0700

> earlier today. Is that is to be expected in an emergency failover?

I restarted the webserver on the backup machine and Pete is now seeing 
correct data.  I'm now proceeding with the failback.  

Glen

---

From: Glen <[email-masked]>
Date: Mon, 5 Sep 2011 20:45:27 -0700

Dear IETF Community -

> back to our Fremont data center at 8:30 PM Pacific Time (4:30 AM GMT) this o

The failback procedure has been completed.  Thank you for your patience.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Glen <[email-masked]>
Date: Mon, 5 Sep 2011 21:55:10 -0700

On Mon, Sep 05, 2011 at 09:06:44PM -0400, Russ Housley wrote:
> Given this situation, it sounds like you made the right call.  You
> might want to leave things running in Orlando until tomorrow.  Your call ...
> Russ

Russ -

Thank you for the message and the support.  I had already started the
reconciliation and failback preparation, and then got caught up with doing
a verification after Pete's trouble report, so I didn't reply to this
earlier.

I did speak with PG&E crews after the fact, and they indicated that the
problem was a failed part "like a breaker" (apparently I don't get to hear
their secret technical terms) and they fixed it and all was well.  So,
at the risk of jinxing everything, we do not expect any more issues, and I
felt it would be best with North America "coming back online" tomorrow to
have us back on the primary facility as quickly as possible. 

Judging that nobody was really doing anything (as confirmed by the marked
lack of activity prior to the failover period) except for Pete (who is
apparently, like me, someone who works 24/7/365), I felt now would be a
more optimal time to fail back, and so proceeded.

I hope this is sufficient.  Please let me know if you have any questions.

Thanks,
Glen

---

From: Russ Housley <[email-masked]>
Date: Tue, 6 Sep 2011 08:12:07 -0400

Seems like all is well.

Russ