October 6th Outage - Post Mortem 1 week ago

I held off writing what I am going to call going forward the “post mortem” report on the outage that affected our network. Magnus Box utilizes third-party storage providers that are HIPAA compliant. We are currently using three of them at the moment to power the US and the EU storage servers. One of our storage providers that powers our US-East storage vault had a massive outage that went on for around 4-5 days. During this time, backups were failing or running very slowly. We opened a priority ticket with them as soon as we noticed that data was flowing into our storage server but not out to the vault. We maintained communication with them over the outage period receiving reports every few hours.

The US-West and EU servers were not affected by this outage, and we posted numerous updates throughout the event. This is the second time this year that we have had a disruption of this magnitude. I want to reassure everyone that we hold our vendors to the highest standards and perform extensive due diligence before we partner with any vendor. Not to mention the level of HIPAA compliance that we require from our vendors. After the fourth/fifth day, the data started flowing through to the storage vault again.

Here is a list of items that we changed or noticed from the outage:

  • We needed a central location to send and post notifications.
    • We posted continuously to Facebook and sent out an alert via MailChimp. However, we discovered that some of our partners are not on Facebook. Because of this, we implemented a new status page powered by https://status.io. You can find it here at https://magnusbox.status.io. I encourage everyone to sign up for notifications. We will be using this system going forward for any outages or maintenance alerts.
  • Transparency is everything.
    • We were very transparent on everything, and even though it was frustrating, we made sure our partners knew what was going on.

Please feel free to reach out to me with any questions.

-Mike Slodowski
Magnus Box LLC,

No Replies on October 6th Outage – Post Mortem

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>