Comparison of forecasts and observations

There are several factors to consider when doing a comparison:

1) Period of data sample: The monthly periods may be too short to be representative, or may represent conditions under one dominant flow pattern that might even be uncommon for that time of year. An anomalously strong eastern trough was present during many of these months. The annual plots will mask seasonal variations.

2) Grid resolution: Peak precipitation amounts will be higher on a finer grid, particularly in summer when spatial variability is greater. Thus the frequency of forecast or observed amounts exceeding a given threshold will be higher on the better resolution grid. Therefore, a fair comparison would be the AVN, the CPC analysis on a 1-degree grid, and the 90-km Eta. The CPC analysis on the 1/4-degree grid will more closely resemble the number of times individual stations or groups of nearby stations in your forecast area exceed the threshold amounts. Remember, the model is forecasting an average over the grid box, so a perfect forecast will be for lower amounts than the peak station values.

3) Model behavior as a function of amount: The model bias will change going from lighter to heavier amounts and this will be different for the Eta and AVN. Compare each threshold separately.

4) Areal coverage or scatter of events vs. frequency of events: A broad region covered with low values and a much smaller region with higher values may both indicate an event of the same frequency. In one case it happens sometimes in one spot, sometimes in another, while in the other case it happens repeatedly over the same spot, missing the surrounding locations.

5) Color scale is exponential! The steps between different colors is bigger as you go up the scale from blue to green to yellow to orange to red to brown. Think of it as a multiplier, a factor of two difference for every two color steps on the scale.

A few particularly noteworthy results:

• AVN produced 1.5" or greater amounts far more frequently than the Eta. Some months the Eta produced more 0.5" events than the AVN.
• Where 1" amounts were observed most frequently, the AVN predicted 1" events the right number of times overall through the year, but elsewhere it predicted such events too often.
• The Eta was better than the AVN at predicting the frequency of occurrence of 0.5" events overall throughout the year.
• Over the data period used, the Eta predicted 2.5"+ amounts far too seldom, but it did predict some in the Los Angeles area and there were some observed there.
• The AVN predicted 2.5"+ amounts in far too many places (though not in the Los Angeles area) and too many events overall. But it didn't predict 2.5"+ events as frequently as observed in spots where the most such events were observed. This suggests that AVN forecasts of 2.5"+ have hardly any skill, not indicating whether or not such heavy amounts covering areas more than 90 km x 90 km may potentially happen or not.
• The frequency of 22 km and 32 km operational Eta model heavy rain events was generally not higher, even during summer, for amounts averaged over 90 km x 90 km areas than for amounts averaged over 32 km x 32 km areas. This suggests that the operational Eta makes its convective rain events over too broad an area. In contrast, the CPC analysis showed cosiderably more frequent heavy rain events in summer when amounts were averaged over 1/4-degree squares than averaged over 1-degree squares, indicating a number of heavy rains occurred on scales between 1/4 degree and 1 degree.
• The Eta predicted 0.5"/24 hr rain events too often in the southeast U.S. during summer.
• Over much of the data period in many regions, the AVN produced slightly more 24-hour and 48-hour accumulated heavy precipitation events during forecast hours up to 66 hours than for forecast hours 72 and beyond.

Remember, model prediction of precipitation is less accurate than model prediction of synoptic weather features. Your precipitation forecast should usually rely on forcings, moisture availability, pre-convective instability and cap, and so on, rather than primarily on the model prediction of precipitation, especially when convection is involved. The purpose of this Web site is to give an indication of whether or not the model QPF may be indicating a rare event or may be a spurious forecast and whether it tends to forecast events of a particular magnitude too often or not often enough.