Hello Fellow ChartWatchers,
We're taking a break from our on-going Technical Analysis 101 series to give you an update on the two disruptions that happened last week. I want to make sure everyone understands what happened and what we are doing to prevent it from happening again.
In case you missed it, on Wednesday, September 9th, our main connection to the Internet was cut. Around 4:45pm Eastern, "some guys" working in a manhole cover outside our offices damaged our fiber cable which knocked us off the air. (I actually spotted those workers as soon as the problem occurred and I went over as asked them if they had touched our cable. Unfortunately, they lied to me and said that they didn't.) Compounding things, the true nature of the problem wasn't obvious for several hours and we had to try and eliminate several alternate theories before we discovered that the cable had in fact been damaged.
After nine hours of testing various theories, we gave up on the fiber cable and moved our site back onto the four T3 connections that - luckily - we still had available. As a result of the nine-hour outage, we gave existing members a free week of additional service.
The following Monday, our index data vendor ThomsonReuters stopped providing us with data for the S&P 500 index along with about 10 other important CBOE-based indexes. The problem was traced down quickly, but Thomson did not re-enable that data for us until after the market closed.
Both of these issues are completely unacceptable. Here's what we are doing to prevent them in the future.
1.) The Gigabit Fiber connection has been fixed. We hope to move our Internet traffic back onto that larger, faster connection Monday evening.
2.) We have ordered a second Gigabit Fiber connection that uses a different physical path so that if one connection gets damaged, the other will continue to work.
3.) We are installing additional physical protection devices for our cables down in the conduits next to our building. We are also trying to track down "those guys" who caused the problem and recover some of our costs.
4.) We have purchased a second router and fiber-optic interfaces to act as backups for our primary equipment.
5.) We are moving our primary data source for CBOE index data from ThomasReuters to IDC/Comstock this week. We are also expediting our entire move off ThomsonReuters as a primary data provider although that process will still take time.
6.) We have sent letters of protest to ThomsonReuters and CBOE about their vague and contradictory communication policies. Unfortunately, we don't have much leverage with those huge companies - which is part of the core problem.
When we do have disruptive problems like this, we will try to communicate as much information about them to you as we can via our Status Blog. Make sure to check that blog whenever you experience a problem accessing our charts.
