Techniques in Hydrologic Forecast Verification

• 1.0 Introduction
  • Objectives
  • Recommended Prerequisite
  • RFC Case Study Examples
  • IVP and EVS Case Studies
  • IVP: Deterministic Approach
  • EVS: Probabilistic Approach
• 2.0 IVP Case Study: OHRFC
  • Basins Studied
  • Forecast Scenarios
  • Basin Aggregation Assessment
    • Slow-response Basin Aggregation
    • Aggregation Outlier
    • Medium-response Basin Aggregation
    • Fast-response Basin Aggregation
    • Compare Fast- and Slow-response Bias
  • Forecast Verification: Errors and Bias
    • MAE for Forecast Scenarios
    • ME (bias) for Forecast Scenarios
  • Correlation and Lead Time
    • Correlation Over All Lead Times
    • Correlation: Short vs. Long Lead Time
    • Correlation Plots
  • Forecast Skill and Lead Time
  • QPF Influence
    • NPVU
    • QPF Impact with IVP
  • Summary of OHRFC Case
  • Review Questions
• 3.0 EVS Case Study: MARFC
  • Basin and Data Characteristics
    • The Data
    • Sample Sizes
  • Verification Scores
    • Data Distribution
    • Mean Continuous Ranked Probability Score (MCRPS)
    • MCRPSS (Skill Score)
    • Conditional Forecast Verification
    • Confidence in Verification
  • Examining Ensemble Means
    • RMSE at Headwater
    • RMSE of QPF Input
    • Mean Error (Bias) of QPF input
    • Mean Error (Bias) of Flow Forecasts
    • Correlation of QPF Input
    • Correlation for Flow Forecasts
  • Ensemble Spread: Box Plots
    • EVS Box Plots
    • Headwater Box Plot: 6-Hour Lead
    • Headwater Box Plot: 30-Hour Lead
    • Headwater Box Plot, Low Flows, Hour 30
    • Headwater Box Plot Questions
    • Headwater Box Plot: 78 hour Lead
    • Headwater Box Plot: 126 Hour Lead
    • Headwater Box Plot Zoomed: 126 Hour Lead
    • Downstream vs. Headwater: 30 Hour Lead
    • Downstream vs. Headwater � 78 Hour Lead
  • QPF Box Plots
    • QPF Box Plots � 24 Hour Lead
    • Climatologically-based QPF Box Plots
  • Mean Continuous Ranked Probability Score (MCRPS)
    • MCRPS of Flow Forecasts
  • Mean Continuous Ranked Probability Skill Score (MCRPSS)
    • Skill of Flow Forecasts
    • Skill with Respect to Lead Time
  • Forecast Reliability: MCRPS
    • Headwater Forecast Reliability
    • Downstream Forecast Reliability
    • QPF Reliability
  • Forecast Reliability: Spread-Bias Diagram
    • Overforecasting on the Spread-bias Diagram
    • Observations Outside the Forecast Distribution
    • Underforecasting on the Spread-bias Diagram
    • Spread-bias Diagram: Lead Hour 6
    • Spread-bias Diagram: Lead Hour 30
    • Spread-bias Diagram: Lead Hour 78
    • Spread-bias Diagram: Lead Hour 126
  • Forecast Discrimination: ROC
    • ROC Curve for QPF
    • ROC Curve for Climatologically-based QPF
    • ROC Score: QPF
    • ROC Scores: Flow Forecasts
  • Confidence in Scores
  • Review Questions
• 4.0 Verification Techniques Summary
  • Meaningful Verification
  • Interpreting Results: OHRFC
  • Interpreting Results: MARFC
  • Summary and Resources

MCRPSS (Skill Score)

For forecast skill we will examine the skill score form of the MCPRS, the MCRPS Skill Score (MCRPSS). Skill scores give us a measure of relative error.

Skill scores are important because it is possible for a forecast system to produce forecasts that are skillful even though they have error.