Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Learn about how we conduct quantitative assessments for observing systems

Our Methodology

OBSERVING SYSTEMS EVALUATIONS

QOSAP conducts quantitative assessments for NOAA, primarily by evaluating observing system experiments (OSEs) and observing system simulation experiments (OSSEs), to provide information on the impact of an alternative mix of current and/or proposed instruments by analyzing tradeoffs (i.e., coverage, resolution, accuracy, and data redundancy) and by optimizing data assimilation and modeling strategies. Costs of developing, deploying, and maintaining new space-based architectures typically exceed $100-500M per instrument.

As new observing systems for Earth Systems modeling applications are proposed, a rigorous evaluation of their potential impact in global and regional numerical weather prediction systems is necessary.  This allows for development of a cost-benefit analysis for better planning and decision making. 
What is an OSE?
OSEs address today’s modeling and observing systems capabilities. They are data denial studies using real data (not simulated) that allow the evaluation of existing data. For example, an OSE may be to run one numerical weather forecast experiment with all the current observations and compare that with a forecast where one type of data is removed from the experiment. The difference in model skill between both experiments indicate the impact of that one data type (one that was removed).
What are the benefits of an OSE?
OSEs help optimize the use of current observations in current modeling systems (i.e., enhanced data assimilation strategies, more realistic characterization of observations, and leverage existing observations not currently utilized).
Limitations of OSEs
However, OSEs cannot be used to analyze the value of enhancing the current global observing system with additional observations that do not yet exist. To quantify the impact of proposed observing capabilities, OSSEs are needed.
Questions OSEs Can Address
Can we do better by optimizing the use of current observations in current modeling systems?

  • Enhanced data assimilation strategies
  • More realistic characterization of observations
  • Management of large volume of data
  • Timeliness for model upgrades
Can we leverage existing observations not currently utilized?

  • Driven by requirements and priorities
  • Investment in personnel and HPC resources
Previous slide
Next slide
OSE vs. OSSE
OSSEs are modeling experiments used to perform an objective evaluation of the potential benefits of proposed instruments. OSSEs are still data denial studies, but they differ from OSEs in that all the observations (current and proposed) are simulated. The impact of one given data type is still investigated by comparing the difference in skill between an experiment with all the observations and one without the proposed data type under investigation. However, a simulated truth is used to simulate all the observations and to quantify the skill of the experiments.
Informing Economic Efficiency
As we know, costs of developing, deploying, and maintaining new space-based architectures typically exceed $100-500 million/instrument. Therefore, there is a need to provide quantitative information on the impact of alternative mix of current and/or proposed instruments to help inform major decisions.
A Platform for Cost / Benefit Analysis
OSSEs provide an ideal platform for this by analyzing tradeoffs (coverage, resolution, accuracy, and data redundancy) and by optimizing data assimilation and modeling strategies. Ultimately, OSSEs help evaluate whether the costs of observing system deployment are justified by the benefits.
Questions OSSEs Can Address
Realism and interpretation of OSSE results:

  • Will the study be completed in time to be useful?
  • Is the predictability of the forecast model realistic?
  • Are the coverage and error characteristics of simulated observations realistic?
  • Are the forecast accuracy of the model and impacts of existing observing systems in the OSSE comparable to the real world?
  • Have the limitations of the OSSE system been determined? (Conclusions should not be drawn beyond these limitations)
Are the costs of observing system deployment justified by the benefits?

Challenges:
  • Computer Power
  • Breadth of information (a lot of pieces in play)
  • Resilience of the overall observing system
  • Statistics vs. individual weather events)
  • Choosing the right verification metrics
Previous slide
Next slide

Types of observations that QOSAP evaluates

Buoys & Ship-Based Observations

Satellite Observations

Ocean and Ecosystem Health Observations

Interact with each image to learn key differences of both methods.

Observing System Experiments (OSEs)

Observing System Experiments (OSEs)

Observing System Experiments (OSEs) evaluate the impact of existing data from deployed instruments on a given model. OSEs can help scientists to determine the most effective way to use observations in a weather model.
Image: First, observations from the real atmosphere are assimilated into a weather model. Next, an analysis is obtained through data assimilation and the skill of subsequent analyses and forecasts is evaluated against subsequent analyses.

Observing System Simulation Experiments (OSSEs)

Observing System Simulation Experiments (OSSEs)

Observing System Simulation Experiments (OSSEs) assess the impact of a potential observing system in a weather model by testing the data from the proposed instrumentation on a simulated environment.
Image: First, synthetic observations are generated from a simulated atmosphere (nature run), and synthetic observations (both existing and proposed) are assimilated into a weather model. Next, an analysis is obtained through data assimilation and the skill of subsequent analyses and forecasts is evaluated against the nature run.

List of selected publications:  Quantitative assessment Papers

Scroll to Top