Fiber Circuit Outage #2 Update #2 - 10/7/2009

I wanted to let everyone know where things stand with our Internet connections after last month's dual whammy.

The fiber cable that was damaged is fully repaired and working well.  We've had our vendor add additional physical protection to the cable itself down in the manhole where the problem occurred.  We don't expect a repeat of that particular problem as a result of those changes.

The Cisco router that also failed on us has been replaced.  The new router is installed and is connected to the fiber cable using completely new hardware.  As of late yesterday afternoon, the entire Fiber Circuit (i.e. cable + router) was working.

As I mentioned before however, we are not going to switch the site traffic back onto the Fiber Circuit for a couple more days.  We do not want a repeat of last time where we quickly switched back only to have it fail again.

Instead, we are going to be testing the circuit and installing an automatic fail-over system.  That work should be done by the middle of next week.  When completed we will be able to switch back on to the Fiber Circuit with confidence that we won't have another site outage.  If the Fiber Circuit fails again for any reason, the automatic fail-over will immediately move our site traffic onto the older T3 circuits that we are using now.

We are also making progress on getting a second Fiber Circuit installed via a completely different physical path.  That will give us fail-over capabilities without the limitations of the older T3 circuits.  That new cable should be installed sometime next month.

As a reminder, we are currently still using those older, slower T3 circuits for the site.  Those 4 circuits have a combined bandwidth of 180 megabits per second.  (The Fiber Circuit can handle 1024 megabits per second.)  Right now, during market hours, those T3 circuits are completely full.  That means that you may experience some slowness during the day - especially around the opening and closing bells.  Please help us out by trying to reduce the number of charts you use during the day whenever possible.

Looking back at our Status blog, so far this year we've endured the following "once in a lifetime" events:

• A major fire and power outage at our ISP's main Seattle facility
• The failure of a critical Dell server
• An idiot in our manhole stomping and breaking our fiber cable
• Our data vendor changing the S&P 500 symbol without warning
• A complete failure of a top-over-the-line Cisco router

Let's hope that our luck improves.  I, on the other hand, will continue to work on improving our infrastructure so that we can better survive these (supposedly) rare kind of issues.