U of Illinois Authors: No ‘Smoking Gun’ Found in Study of USDA Reports

via a special arrangement with Informa Economics, Inc.

Study offers suggestions for the future, but no evidence why traders have badly missed some USDA data

No “smoking gun” was found in an exam of USDA reports to determine why trade expectations have badly missed the actual data published by USDA, according to a study released today by the University of Illinois. Link to report.

Noting that the substantial difference between trade expectations and the actual USDA data published started in 2006, the report looked at a number of potential explanations for the decline offered by various market participants.

Researchers used the four following criteria in examining the issues involved:

• Why corn and not soybeans?

• Why 2006-2012 and not earlier?

• Why only in particular marketing years? and

• Why a pattern of reversals during marketing years?

Researchers said “all but one of the potential explanations offered to date clearly fails to satisfy at least one of the criteria,” the report stated. “The explanation with the most merit is that unresolved errors in production estimates for corn led to the large surprises. NASS stocks estimates undoubtedly encompass sampling errors for both production and stocks estimates and it is likely that unresolved sampling errors for corn production estimates are large enough to explain the surprises.”

Their findings included the following relative to the exam of the World Ag Outlook Board (WAOB) and National Ag Statistics Service (NASS):

• WAOB corn and soybean yield forecasts made in May, June, and July do not have a substantial bias. The accuracy of the forecasts also has not changed markedly over the 1993-2012 time period for either corn or soybeans. With a few exceptions, WAOB corn and soybean forecast errors since 2006 generally are within the historical range of errors.

• NASS yield forecasts for corn reveal no evidence of bias in any month over 1990-2012 and forecast errors since 2006 are well within the historical range of errors. There is some evidence of improvement in the accuracy of NASS corn yield forecasts over time. Soybean forecast errors since 2006 are also within the historical range, except for September and October 2012, and there is no statistical change in the magnitude of forecast errors for soybean yields over time. However, there is a general tendency for soybean forecasts to be conservative, in the sense of underestimating final yield. In addition, market analysts consistently under-estimated NASS production forecasts during the first half of the sample and over-estimated production forecasts during the second half.

• There has been a sharp decline in market analysts’ ability to anticipate quarterly corn usage as implied by NASS Grain Stocks reports since the start of the 2006 marketing year. Double-digit implied usage surprises occurred three times during 2006-2012 and each one substantially exceeded the largest surprise observed over 1990-2005. The most problematic corn stocks estimates occurred in the 2009, 2010, and 2012 marketing years. Within these three problematic years there was also a clear tendency towards reversal of the surprises from quarter-to-quarter.

The authors conclude that it is “more difficult to pin down exactly why unresolved sampling errors for corn production estimates were concentrated in 2009, 2010, and 2012 and caused the quarter-to-quarter reversal pattern in surprises, but reasonable arguments can be put forward. Nonetheless, it is important to emphasize there is no “smoking gun” in terms of the available evidence on the impact of unresolved production sampling errors, with some of the evidence best described as circumstantial.”

Based on the analysis by researchers, they offer the following recommendations regarding USDA corn and soybean forecasts and estimates:

• The WAOB should describe in a written document the exact process used to determine corn and soybean yield forecasts for each month, including the roles of crop weather regression forecasts, subjective judgment, and any other inputs, and this document should be available on the WAOB website and explicitly referenced and hyperlinked in the footnotes of the relevant supply and demand tables in May, June, and July World Agricultural Supply and Demand Estimates (WASDE) reports.

• NASS should institute an internal review of soybean yield forecasting procedures to determine the source of any bias and make changes needed to insure it is eliminated.

• NASS should “open up the black box” for each monthly corn and soybean yield forecast as much as possible. This should include: i) presentation of state and national yield forecasts derived from the agricultural yield survey (AYS) and the objective yield survey (OYS), as well as the usual composite forecast derived from the two surveys, ii) presentation of assumptions regarding fruit weights used in deriving OYS yield forecasts during forecast months when these measurements are not available, and iii) some form of recognition of the degree to which weather and crop condition data influence composite forecasts.

• NASS should initiate a research project to study how yield monitor data could be incorporated into crop yield estimation procedures.

• NASS should initiate an internal review of corn stock estimation procedures in an effort to determine whether any methodological problems are apparent.

• NASS should provide the same instructions regarding weight per bushel that it provides to off-farm survey respondents to on-farm survey respondents.

• NASS should investigate the possibility of adding grain stocks questions to the Agricultural Census.

• NASS should engage market participants in a discussion of the appropriate interpretation of grain stocks estimates and consider what means might be available to improve the general understanding of the limits of stock estimates for implying usage.

• WAOB and NASS should evaluate the potential costs and benefits of adding a survey of corn feed use that would allow a fuller accounting of corn usage similar to what has been historically possible for soybeans.

• WAOB and NASS should investigate the potential costs and benefits of adding a survey of ethanol plants to provide more accurate estimates of corn used in ethanol production.

• WAOB and NASS should seek funding to replace the former monthly Census Bureau M311J Fats and Oils: Oilseed Crushings report.

  ---

Comments: The report appears to echo USDA’s own findings when they have looked at this issue – they can find no one single reason why the trade expectations have missed the actual data. However, this at least provides another analytical look at the USDA data process and also provides some suggestions that still may not resolve the questions that have been raised over the last several years about the USDA data. Some readers have asked to find out how much this study cost and who exactly got the money. I will follow-up on those questions.

There are several suggestions in the report for USDA and at least one will already be answered -- the NASS now taking over doing the Industrial Reports that had been discontinued by the Census Bureau. USDA officials are also poring over the report to assess the recommendations but sources we’ve talked with already signal some “relief” but perhaps “disappointment” with the report in that they did not point to one factor to explain the issues University of Illinois analysts were asked to explore.

UPDATE... A reader reacts:

“Excuse me. Can’t explain why traders missed the USDA numbers by so much? Please have them explain why implied corn feed/residual jumps wildly quarter to quarter with no relationship to animal units, while marketing year data is calm/tame by comparison.

“As I recall in early June 2013…Why did WASDE June 2013 sharply lower their OLD CROP feed/residual use WITHOUT a stocks report to support the change…

4400 mil bu versus 4600 in May so -200 mil bu.

And then on July 2013 WASDE (4650) they had to sharply increase their feed use because the USDA WASDE badly missed the June 30 stocks estimate.”