Geospatial computation, data transformation to a relevant statistical software, and step-wise quantitative performance assessment can be cumbersome, especially when considering that the entire modelling procedure is repeatedly interrupted by several input/output steps, and the self-consistency and self-adaptive response to the modelled data and the features therein are lost while handling the data from different kinds of working environments. To date, an automated and a comprehensive validation system, which includes both the cutoff-dependent and –independent evaluation criteria for spatial modelling approaches, has not yet been developed for GIS based methodologies. This study, for the first time, aims to fill this gap by designing and evaluating a user-friendly model validation approach, denoted as Performance Measure Tool (PMT), and developed using freely available Python programming platform. The considered cutoff-dependent criteria include receiver operating characteristic (ROC) curve, success-rate curve (SRC) and prediction-rate curve (PRC), whereas cutoff-independent consist of twenty-one performance metrics such as efficiency, misclassification rate, false omission rate, F-score, threat score, odds ratio, etc. To test the robustness of the developed tool, we applied it to a wide variety of geo-environmental modelling approaches, especially in different countries, data, and spatial contexts around the world including, the USA (soil digital modelling), Australia (drought risk evaluation), Vietnam (landslide studies), Iran (flood studies), and Italy (gully erosion studies). The newly proposed PMT is demonstrated to be capable of analyzing a wide range of environmental modelling results, and provides inclusive performance evaluation metrics in a relatively short time and user-convenient framework whilst each of the metrics is used to address a particular aspect of the predictive model. Drawing on the inferences, a scenario-based protocol for model performance evaluation is suggested.

Rahmati, O., Kornejady, A., Samadi, M., Deo, R.C., Conoscenti, C., Lombardo, L., et al. (2019). PMT: New analytical framework for automated evaluation of geo-environmental modelling approaches. SCIENCE OF THE TOTAL ENVIRONMENT, 664, 296-311 [10.1016/j.scitotenv.2019.02.017].

PMT: New analytical framework for automated evaluation of geo-environmental modelling approaches

Conoscenti, Christian;
2019-01-01

Abstract

Geospatial computation, data transformation to a relevant statistical software, and step-wise quantitative performance assessment can be cumbersome, especially when considering that the entire modelling procedure is repeatedly interrupted by several input/output steps, and the self-consistency and self-adaptive response to the modelled data and the features therein are lost while handling the data from different kinds of working environments. To date, an automated and a comprehensive validation system, which includes both the cutoff-dependent and –independent evaluation criteria for spatial modelling approaches, has not yet been developed for GIS based methodologies. This study, for the first time, aims to fill this gap by designing and evaluating a user-friendly model validation approach, denoted as Performance Measure Tool (PMT), and developed using freely available Python programming platform. The considered cutoff-dependent criteria include receiver operating characteristic (ROC) curve, success-rate curve (SRC) and prediction-rate curve (PRC), whereas cutoff-independent consist of twenty-one performance metrics such as efficiency, misclassification rate, false omission rate, F-score, threat score, odds ratio, etc. To test the robustness of the developed tool, we applied it to a wide variety of geo-environmental modelling approaches, especially in different countries, data, and spatial contexts around the world including, the USA (soil digital modelling), Australia (drought risk evaluation), Vietnam (landslide studies), Iran (flood studies), and Italy (gully erosion studies). The newly proposed PMT is demonstrated to be capable of analyzing a wide range of environmental modelling results, and provides inclusive performance evaluation metrics in a relatively short time and user-convenient framework whilst each of the metrics is used to address a particular aspect of the predictive model. Drawing on the inferences, a scenario-based protocol for model performance evaluation is suggested.
2019
Rahmati, O., Kornejady, A., Samadi, M., Deo, R.C., Conoscenti, C., Lombardo, L., et al. (2019). PMT: New analytical framework for automated evaluation of geo-environmental modelling approaches. SCIENCE OF THE TOTAL ENVIRONMENT, 664, 296-311 [10.1016/j.scitotenv.2019.02.017].
File in questo prodotto:
File Dimensione Formato  
2019 - Rahmati et al. - Science of The Total Environment - PMT New analytical framework for automated evaluation of geo-environmental mo.pdf

Solo gestori archvio

Tipologia: Versione Editoriale
Dimensione 4.62 MB
Formato Adobe PDF
4.62 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
STOTEN-D-18-13868R1.pdf

accesso aperto

Tipologia: Post-print
Dimensione 3.97 MB
Formato Adobe PDF
3.97 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10447/341632
Citazioni
  • ???jsp.display-item.citation.pmc??? 4
  • Scopus 82
  • ???jsp.display-item.citation.isi??? 80
social impact