Commodity Data at StockCharts - Past, Present, and Future

Today we removed the Open, High and Low data from many of our commodity based indexes like $SILVER and $USD.  Those charts now appear as line charts connecting individual closing points instead of the bar or candlestick charts people saw before.

Some people are not happy with us for making this change.  I understand that frustration.  If there were some other way to fix the issue, we would have done that instead.  Here are the reasons for removing that data:

The number one reason for removing that data is because after a long investigation, we have lost faith in the accuracy of those data points.  As we completed our transition off of the ThomsonReuters datafeed and onto the IDC datafeed, we were forced to take a hard look at the "continuous contract" data we've been receiving from Thomson for years.  In the course of comparing that data to other sources, we noticed large numbers of discrepancies with the Open, High and Low data points.  (The Close data points were reliable however.)

Drilling down into the raw data, we saw that many of the problems occurred because the data was inconsistent with itself.  By that I mean that the closing value was lower than the low or the closing value was higher than the high.  Our system automatically adjusts for those inconsistencies so that crazy looking candles don't show up on the charts but the bar and candlesticks on those charts were often inaccurate.

For example, if we saw an End-of-Day quote where the Open was 10, the High was 11, the Low was 9, and the Close was 7 (i.e. lower than the low) what should we do?  Do we move the Low down to 7?  Do we move the Close up to 9?  Do we throw it all out?  It's pretty much a lose-lose situation regardless of what we choose.

Unfortunately, what we discovered this week is that lots of the Open, High and Low values for our EOD commodity based indexes were having that exact problem occur frequently and the problem had been going on for some time.

As long-term subscribers know, accuracy is a big deal with us here at StockCharts.  We work very hard to provide data and calculations that are as accurate as possible.  Faced with the reality that large amounts of our EOD commodity data was incorrect, I decided to have it removed rather than continue to potentially mislead people.

That difficult decision was made easier by the fact that we have always been up-front with people about the nature of our commodity based indexes.  Specifically, our support area has contained the following warning for years:

"We do provide end-of-day charts of certain commodity indexes.  Those charts are based on the non-tradeable "continuous" contract for each commodity.  They are designed to help equity traders understand the mid- and long-term directions of various commodities for help with making decisions about trading equities.  They are not designed for actual futures trading."

We have been very careful over the years to never bill ourselves as a commodities web site.  We are an equities web site.  We are not FuturesCharts.com; we are StockCharts.com.  We do not have access to a full commodities data feed and we have never provided charts or data for actual commodities contracts.

Given those warnings and the end-of-day nature of our commodity charts, I was hopeful that this issue would not affect many people.  Given the proper mid-to-long-term perspective, a close-only chart and a OHLC chart provide you with the same general information about the direction and performance of an index.

For example, many people do what John Murphy suggests and use Intermarket Performance Charts and Ratio charts to compare indexes.  Both of those charting styles focus on closing values, not Highs and Lows and so that kind of analysis is not affected by this change at all.

The problems occur when people made incorrect assumptions about what our commodity charts were based on and began using them to make commodity trading decisions.  To those people I say the same thing now that I've always said about this matter:  Don't use our EOD commodity charts for making futures trading decisions!

In many ways I believe that removing the Open, High, and Low from those charts is a good thing because it re-reinforces the purpose of those charts and minimizes the chances that they will be misused in the future.

Speaking of the future, what comes next?

It has always been StockCharts.com's intention to add legitimate, tradable futures data to our website at some point.  It is still our intention to do that.  The economics of acquiring and redistributing that data legally is still very challenging however and so we are still unable to provide real futures charts at this time.

That said, we are continuing to look for possible "middle-ground" solutions which would allow us to provide, for example, delayed spot prices for popular commodities.  (The great irony here is that the search for better commodity data is what led us to discover the bad OHL data in our current indexes.)  At this point, we still have not found a solution for commodities data beyond what we currently provide but we are continuing to look.

In summary:

1. The old commodity data for Open, High and Low was incorrect and has been removed.
2. Our commodity indexes shouldn't be used for trading.
3. You can still do trend, performance and ratio analysis of our current commodity charts which is always how we intended them to be used.
4. Better commodity data will be available at some point in the future but we do not know when.

 

- Chip