1
|
Canbek G. BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems. INT J MACH LEARN CYB 2023:1-31. [PMID: 37360884 PMCID: PMC10113998 DOI: 10.1007/s13042-023-01826-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2022] [Accepted: 03/21/2023] [Indexed: 06/28/2023]
Abstract
Probabilistic error/loss performance evaluation instruments that are originally used for regression and time series forecasting are also applied in some binary-class or multi-class classifiers, such as artificial neural networks. This study aims to systematically assess probabilistic instruments for binary classification performance evaluation using a proposed two-stage benchmarking method called BenchMetrics Prob. The method employs five criteria and fourteen simulation cases based on hypothetical classifiers on synthetic datasets. The goal is to reveal specific weaknesses of performance instruments and to identify the most robust instrument in binary classification problems. The BenchMetrics Prob method was tested on 31 instrument/instrument variants, and the results have identified four instruments as the most robust in a binary classification context: Sum Squared Error (SSE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE, as the variant of MSE), and Mean Absolute Error (MAE). As SSE has lower interpretability due to its [0, ∞) range, MAE in [0, 1] is the most convenient and robust probabilistic metric for generic purposes. In classification problems where large errors are more important than small errors, RMSE may be a better choice. Additionally, the results showed that instrument variants with summarization functions other than mean (e.g., median and geometric mean), LogLoss, and the error instruments with relative/percentage/symmetric-percentage subtypes for regression, such as Mean Absolute Percentage Error (MAPE), Symmetric MAPE (sMAPE), and Mean Relative Absolute Error (MRAE), were less robust and should be avoided. These findings suggest that researchers should employ robust probabilistic metrics when measuring and reporting performance in binary classification problems.
Collapse
|
2
|
Prediction of severe thunderstorm events with ensemble deep learning and radar data. Sci Rep 2022; 12:20049. [PMID: 36414648 PMCID: PMC9681835 DOI: 10.1038/s41598-022-23306-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Accepted: 10/29/2022] [Indexed: 11/23/2022] Open
Abstract
The problem of nowcasting extreme weather events can be addressed by applying either numerical methods for the solution of dynamic model equations or data-driven artificial intelligence algorithms. Within this latter framework, the most used techniques rely on video prediction deep learning methods which take in input time series of radar reflectivity images to predict the next future sequence of reflectivity images, from which the predicted rainfall quantities are extrapolated. Differently from the previous works, the present paper proposes a deep learning method, exploiting videos of radar reflectivity frames as input and lightning data to realize a warning machine able to sound timely alarms of possible severe thunderstorm events. The problem is recast in a classification one in which the extreme events to be predicted are characterized by a an high level of precipitation and lightning density. From a technical viewpoint, the computational core of this approach is an ensemble learning method based on the recently introduced value-weighted skill scores for both transforming the probabilistic outcomes of the neural network into binary predictions and assessing the forecasting performance. Such value-weighted skill scores are particularly suitable for binary predictions performed over time since they take into account the time evolution of events and predictions paying attention to the value of the prediction for the forecaster. The result of this study is a warning machine validated against weather radar data recorded in the Liguria region, in Italy.
Collapse
|
3
|
The Reconstitution Predictive Network for Precipitation Nowcasting. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.07.061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
4
|
WRF Model Sensitivity to Spatial Resolution in Singapore; Analysis for a Heavy Rain Event and General Suitability. ATMOSPHERE 2022. [DOI: 10.3390/atmos13040606] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Granularity of the grid (both horizontally and vertically) is a key consideration when conducting localised Numerical Weather Prediction (NWP) modelling. Generally speaking, an NWP model with a finer grid can explicitly resolve more processes and require less parameterisation. However, a finer grid also requires more computation and it is not always clear that a finer grid will produce a more accurate forecast. In this study, we explore the sensitivity of rainfall prediction over Singapore to grid resolution. We use the Weather and Research Forecasting model (WRF) to forecast rainfall over Singapore and explore the performance of horizontal resolutions ranging from 1 km to 12 km. We test the performance on a set of dates from across the years 2020–2021 against both ground observations and radar-derived rain rates. When compared to ground observations, we show that, overall, the higher resolution produces the highest Critical Success Index (CSI) for rain rates in excess of 0.5 mm/h. When compared against radar-derived rain rates, the 1 km domain produces superior CSI values for all rain rates. The daily-average hourly Fractional Skill Score (FSS) was then calculated for some dates and showed agreement with the CSI results with the exception of a north-east monsoon day where, for heavier rain rates, the 3 km domain has superior FSS. We also investigate a particularly heavy rain event on 10 January 2021 and show that the 3 km grid has highest CSI for rain rates of 4 mm/h and 16 mm/h (based on both ground-based and radar-derived rain rates), however, the 1 km has superior FSS for this event.
Collapse
|
5
|
ConvLSTM Network-Based Rainfall Nowcasting Method with Combined Reflectance and Radar-Retrieved Wind Field as Inputs. ATMOSPHERE 2022. [DOI: 10.3390/atmos13030411] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/10/2022]
Abstract
Strong convection nowcasting has been gaining importance in regional security, economic development, and water resource management. Rainfall nowcasting with strong timeliness needs to effectively forecast the intensity of rainfall in a local region in the short term. The forecast performance of traditional methods is limited. In this paper, a rainfall nowcasting model based on the Convolutional Long Short-Term Memory (ConvLSTM) is proposed. Combined reflectance (CR) and the retrieved wind field are selected as the key factors for prediction. The model considers the influence of water vapor content, transport, and change on rainfall by measuring CR and the retrieved wind field. Forecast results are compared to different models and different input data schemes. The analysis shows that the CSI scores of this method can reach about 0.8, which is 28% higher than that of the optical flow method. The ConvLSTM structure with spatial analysis and time memory can greatly enhance the predictive ability of the model, and the addition of wind field data also improves the forecasting performance of the model. Therefore, the idea of forecasting rainfall on the basis of water vapor content and water vapor transport is practical and effective, and the accuracy of a forecast can be improved by using ConvLSTM network to extract spatiotemporal features.
Collapse
|
6
|
Luo C, Zhao X, Sun Y, Li X, Ye Y. PredRANN: The spatiotemporal attention Convolution Recurrent Neural Network for precipitation nowcasting. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2021.107900] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
7
|
A Novel Multi-Input Multi-Output Recurrent Neural Network Based on Multimodal Fusion and Spatiotemporal Prediction for 0–4 Hour Precipitation Nowcasting. ATMOSPHERE 2021. [DOI: 10.3390/atmos12121596] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Multi-source meteorological data can reflect the development process of single meteorological elements from different angles. Making full use of multi-source meteorological data is an effective method to improve the performance of weather nowcasting. For precipitation nowcasting, this paper proposes a novel multi-input multi-output recurrent neural network model based on multimodal fusion and spatiotemporal prediction, named MFSP-Net. It uses precipitation grid data, radar echo data, and reanalysis data as input data and simultaneously realizes 0–4 h precipitation amount nowcasting and precipitation intensity nowcasting. MFSP-Net can perform the spatiotemporal-scale fusion of the three sources of input data while retaining the spatiotemporal information flow of them. The multi-task learning strategy is used to train the network. We conduct experiments on the dataset of Southeast China, and the results show that MFSP-Net comprehensively improves the performance of the nowcasting of precipitation amounts. For precipitation intensity nowcasting, MFSP-Net has obvious advantages in heavy precipitation nowcasting and the middle and late stages of nowcasting.
Collapse
|
8
|
Broad-UNet: Multi-scale feature learning for nowcasting tasks. Neural Netw 2021; 144:419-427. [PMID: 34563751 DOI: 10.1016/j.neunet.2021.08.036] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2021] [Revised: 05/21/2021] [Accepted: 08/30/2021] [Indexed: 11/21/2022]
Abstract
Weather nowcasting consists of predicting meteorological components in the short term at high spatial resolutions. Due to its influence in many human activities, accurate nowcasting has recently gained plenty of attention. In this paper, we treat the nowcasting problem as an image-to-image translation problem using satellite imagery. We introduce Broad-UNet, a novel architecture based on the core UNet model, to efficiently address this problem. In particular, the proposed Broad-UNet is equipped with asymmetric parallel convolutions as well as Atrous Spatial Pyramid Pooling (ASPP) module. In this way, the Broad-UNet model learns more complex patterns by combining multi-scale features while using fewer parameters than the core UNet model. The proposed model is applied on two different nowcasting tasks, i.e. precipitation maps and cloud cover nowcasting. The obtained numerical results show that the introduced Broad-UNet model performs more accurate predictions compared to the other examined architectures.
Collapse
|
9
|
Zhang F, Wang X, Guan J, Wu M, Guo L. RN-Net: A Deep Learning Approach to 0-2 Hour Rainfall Nowcasting Based on Radar and Automatic Weather Station Data. SENSORS 2021; 21:s21061981. [PMID: 33799726 PMCID: PMC7998606 DOI: 10.3390/s21061981] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/01/2021] [Revised: 03/06/2021] [Accepted: 03/08/2021] [Indexed: 11/16/2022]
Abstract
Precipitation has an important impact on people’s daily life and disaster prevention and mitigation. However, it is difficult to provide more accurate results for rainfall nowcasting due to spin-up problems in numerical weather prediction models. Furthermore, existing rainfall nowcasting methods based on machine learning and deep learning cannot provide large-area rainfall nowcasting with high spatiotemporal resolution. This paper proposes a dual-input dual-encoder recurrent neural network, namely Rainfall Nowcasting Network (RN-Net), to solve this problem. It takes the past grid rainfall data interpolated by automatic weather stations and doppler radar mosaic data as input data, and then forecasts the grid rainfall data for the next 2 h. We conduct experiments on the Southeastern China dataset. With a threshold of 0.25 mm, the RN-Net’s rainfall nowcasting threat scores have reached 0.523, 0.503, and 0.435 within 0.5 h, 1 h, and 2 h. Compared with the Weather Research and Forecasting model rainfall nowcasting, the threat scores have been increased by nearly four times, three times, and three times, respectively.
Collapse
Affiliation(s)
- Fuhan Zhang
- School of Computer, National University of Defense Technology, Changsha 410000, China; (F.Z.); (M.W.); (L.G.)
| | - Xiaodong Wang
- School of Computer, National University of Defense Technology, Changsha 410000, China; (F.Z.); (M.W.); (L.G.)
- Correspondence:
| | - Jiping Guan
- School of Meteorology and Oceanography, National University of Defense Technology, Changsha 410000, China;
| | - Meihan Wu
- School of Computer, National University of Defense Technology, Changsha 410000, China; (F.Z.); (M.W.); (L.G.)
| | - Lina Guo
- School of Computer, National University of Defense Technology, Changsha 410000, China; (F.Z.); (M.W.); (L.G.)
| |
Collapse
|
10
|
Zheng S, Hu X. Early Warning Method for Public Health Emergency Under Artificial Neural Network in the Context of Deep Learning. Front Psychol 2021; 12:594031. [PMID: 33658958 PMCID: PMC7917260 DOI: 10.3389/fpsyg.2021.594031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Accepted: 01/18/2021] [Indexed: 11/30/2022] Open
Abstract
The purpose is to minimize the substantial losses caused by public health emergencies to people’s health and daily life and the national economy. The tuberculosis data from June 2017 to 2019 in a city are collected. The Structural Equation Model (SEM) is constructed to determine the relationship between hidden and explicit variables by determining the relevant indicators and parameter estimation. The prediction model based on Artificial Neural Network (ANN) and Convolutional Neural Network (CNN) is constructed. The method’s effectiveness is verified by comparing the prediction model’s loss value and accuracy in training and testing. Meanwhile, 50 pieces of actual cases are tested, and the warning level is determined according to the T-value. The results show that comparing and analyzing ANN, CNN, and the hybrid network of ANN and CNN, the hybrid network’s accuracy (95.1%) is higher than the other two algorithms, 89.1 and 90.1%. Also, the hybrid network has sound prediction effects and accuracy when predicting actual cases. Therefore, the early warning method based on ANN in deep learning has better performance in public health emergencies’ early warning, which is significant for improving early warning capabilities.
Collapse
Affiliation(s)
- Shuang Zheng
- College of Media and International Culture, Zhejiang University, Hangzhou, China.,School of Media and Law, NingboTech University, Ningbo, China
| | - Xiaomei Hu
- School of Media and Law, NingboTech University, Ningbo, China
| |
Collapse
|
11
|
Enhancing the Encoding-Forecasting Model for Precipitation Nowcasting by Putting High Emphasis on the Latest Data of the Time Step. ATMOSPHERE 2021. [DOI: 10.3390/atmos12020261] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/16/2023]
Abstract
Nowcasting is an important technique for weather forecasting because sudden weather changes significantly affect human life. The encoding-forecasting model, which is a state-of-the-art architecture in the field of data-driven radar extrapolation, does not particularly focus on the latest data when forecasting natural phenomena. This paper proposes a weighted broadcasting method that emphasizes the latest data of the time step to improve the nowcasting performance. This weighted broadcasting method allows the most recent rainfall patterns to have a greater impact on the forecasting network by extending the architecture of the existing encoding-forecasting model. Experimental results show that the proposed model is 1.74% and 2.20% better than the existing encoding-forecasting model in terms of mean absolute error and critical success index, respectively. In the case of heavy rainfall with an intensity of 30 mm/h or higher, the proposed model was more than 30% superior to the existing encoding-forecasting model. Therefore, applying the weighted broadcasting method, which explicitly places a high emphasis on the latest information, to the encoding-forecasting model is considered as an improvement that is applicable to the state-of-the-art implementation of data-driven radar-based precipitation nowcasting.
Collapse
|
12
|
A Novel LSTM Model with Interaction Dual Attention for Radar Echo Extrapolation. REMOTE SENSING 2021. [DOI: 10.3390/rs13020164] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
The task of precipitation nowcasting is significant in the operational weather forecast. The radar echo map extrapolation plays a vital role in this task. Recently, deep learning techniques such as Convolutional Recurrent Neural Network (ConvRNN) models have been designed to solve the task. These models, albeit performing much better than conventional optical flow based approaches, suffer from a common problem of underestimating the high echo value parts. The drawback is fatal to precipitation nowcasting, as the parts often lead to heavy rains that may cause natural disasters. In this paper, we propose a novel interaction dual attention long short-term memory (IDA-LSTM) model to address the drawback. In the method, an interaction framework is developed for the ConvRNN unit to fully exploit the short-term context information by constructing a serial of coupled convolutions on the input and hidden states. Moreover, a dual attention mechanism on channels and positions is developed to recall the forgotten information in the long term. Comprehensive experiments have been conducted on CIKM AnalytiCup 2017 data sets, and the results show the effectiveness of the IDA-LSTM in addressing the underestimation drawback. The extrapolation performance of IDA-LSTM is superior to that of the state-of-the-art methods.
Collapse
|
13
|
Performance Comparison between Deep Learning and Optical Flow-Based Techniques for Nowcast Precipitation from Radar Images. FORECASTING 2020. [DOI: 10.3390/forecast2020011] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
In this article, a nowcasting technique for meteorological radar images based on a generative neural network is presented. This technique’s performance is compared with state-of-the-art optical flow procedures. Both methods have been validated using a public domain data set of radar images, covering an area of about 104 km2 over Japan, and a period of five years with a sampling frequency of five minutes. The performance of the neural network, trained with three of the five years of data, forecasts with a time horizon of up to one hour, evaluated over one year of the data, proved to be significantly better than those obtained with the techniques currently in use.
Collapse
|
14
|
Precipitation Nowcasting with Orographic Enhanced Stacked Generalization: Improving Deep Learning Predictions on Extreme Events. ATMOSPHERE 2020. [DOI: 10.3390/atmos11030267] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
One of the most crucial applications of radar-based precipitation nowcasting systems is the short-term forecast of extreme rainfall events such as flash floods and severe thunderstorms. While deep learning nowcasting models have recently shown to provide better overall skill than traditional echo extrapolation models, they suffer from conditional bias, sometimes reporting lower skill on extreme rain rates compared to Lagrangian persistence, due to excessive prediction smoothing. This work presents a novel method to improve deep learning prediction skills in particular for extreme rainfall regimes. The solution is based on model stacking, where a convolutional neural network is trained to combine an ensemble of deep learning models with orographic features, doubling the prediction skills with respect to the ensemble members and their average on extreme rain rates, and outperforming them on all rain regimes. The proposed architecture was applied on the recently released TAASRAD19 radar dataset: the initial ensemble was built by training four models with the same TrajGRU architecture over different rainfall thresholds on the first six years of the dataset, while the following three years of data were used for the stacked model. The stacked model can reach the same skill of Lagrangian persistence on extreme rain rates while retaining superior performance on lower rain regimes.
Collapse
|