On the Assessment of Daily Equatorial Plasma Bubble Occurrence Modeling and Forecasting

Predicting the daily variability of Equatorial Plasma Bubbles (EPBs) is an ongoing scientific challenge. Various methods for predicting EPBs have been developed, however, the research community is yet to scrutinize the methods for evaluating and comparing these prediction models/techniques. In this study, 12 months of co-located GPS and UHF scintillation observations spanning South America, Atlantic/Western Africa, Southeast Asia, and Pacific sectors are used to evaluate the Generalized Rayleigh-Taylor (R-T) growth rates calculated from the Thermosphere Ionosphere Electrodynamics General Circulation Model (TIEGCM). Various assessment metrics are explored, including the use of significance testing on skill scores for threshold selection. The sensitivity of these skill scores to data set type (i.e., GPS versus UHF) and data set size (30, 50, 60, and 90 days/events) is also investigated. It is shown that between 50 and 90 days is required to achieve a statistically significant skill score. Methods for conducting model-model comparisons are also explored, including the use of model “sufficiency.” However, it is shown that the results of model-model comparisons must be carefully interpreted and can be heavily dependent on the data set used. It is also demonstrated that the observation data set must exhibit an appropriate level of daily EPB variability in order to assess the true strength of a given model/technique. Other limitations and considerations on assessment metrics and future challenges for EPB prediction studies are also discussed.