Interactive tool for model evaluation

Aim

Our goal is to develop an interactive model evaluation tool (MET) to evaluate Species Distribution Models (SDMs) and synthesize expert evaluation with quantitative evaluation to inform model improvement and application.

What are the problems we’re trying to solve?

• Unfair, ambiguity and disperse critique of modeling

• Frustrations and miscommunications on both model users, evaluators and modelers

• Miscommunications with downstream information/opinion-receivers

• Missed opportunities for model improvements

• Loss of model information that could be well used for inputs to actions/decisions due to interactions with model outputs that are due to:

  • Inexplicit modelers intentions’ about the utility and applicability of their model and mismatch with evaluators’ purpose(s)

  • Undisciplined path of model interaction and evaluation

  • Purposes not well aligned with modeling utility

  • Model appropriateness regarded as Yes/No instead of gradient of appropriateness

Approach

We propose integrating multiple modules within an interactive tool, as follow:

Interactive model evaluation tool

This module integrates all modules describe below and facilitates multiple users navigating, inquiring and reacting to model materials.

Model development module

It will allow modelers to upload model inputs, outputs and metadata to the MET.

We are using the ODMAP Shiny app developed by (Zurell et al., 2020) as a template to upload model metadata

We will evaluate SDMs for multiple species from:

• The Ontario Breeding Bird Atlas models (Atlas-3).

However, we will be also testing:

• The upcoming version 5 (V5) models from the Boreal Avian Modelling project (BAM)

Model Evaluation module

It will enable evaluators to provide feedback on model materials. We are using the ODMAP Shiny app developed by (Zurell et al., 2020) as a template tocreate a model evaluation tool.

Reporting module

It will allow evaluator to summarize their evaluation and

Synthesis module

it will combines feedback from multiple evaluators with relevant model materials to inform model use and improvement

MET functionality

• Interactive tool: it will allow multiple users to:

  • Interact with multiple datasets formats (maps, text, tables, etc) in a shiny app.

  • Display multiple panels of information (e.g., comparing predictions for two species).

  • Write comments, suggestions, recommendations (text).

  • Rank model inputs, outputs and metadata

  • Save and submit feedback (option to save it temporarily and resume later).

  • Save strategic information from users and evaluators purposes (option to save it temporarily and resume later)

  • Revisit evaluation

  • Track the path taken by evaluators to provide feedback and extract information.

  • Provide feedback on other evaluators’ feedback

  • Ability to generate reports

• Hosted: the MET will be stored in a Github repository defined by ECCC and a functional version of the app will be set available by the contractor in a website defined by ECCC with some projects for demonstrations (The Ontario Breeding Bird Atlas models (Atlas-3) and the upcoming v5 models from the Boreal Avian Modelling project (BAM).

• Portability: it allows users to install and implement the tool locally on different platforms or environments with minimal changes.

• Upload data sets: it will allow users to populate the MET, by uploading models’ inputs, outputs and metadata for multiple species.

• Compile, store, and synthesize: the tool should allow to compile, store and summarize feedback and actions proposed by multiple users.

• Track and report changes and progress: the MET will provide the functionality to track, consolidate and visualize changes and progress in model evaluation (e.g., species X has been evaluated 3 time by Y, main action include A,B and C, improvement are “spatial predictions” and model performance metrics, etc.). 

MET attributes

• Modular: the MET will be developed using independent modules, each handling a specific functionality (for instance a module to upload metadata, a module to capture evaluators’ feedback etc.). Breaking down the complexity of the tool into modules will help in maintenance, functionality updates, troubleshooting, etc. The contractor will provide the functionality to effectively link these modules.

• Expandable: we want to expand MET by adding new modules that will be only conceptualized in this contract (e.g., model application & risk assessment, etc.)..

• Flexible:  It will also be the frequent case to add items or sections to specific modules of the MET (e.g., a new task that requires feedback from evaluators)

• Generalizable: our vision is that the MET can be used in multiple types of models and multiple projects.

• Spatially explicit: users will have the opportunity to visualize, interact and provide feedback on multiple raster and vector layers (predictions, uncertainty, covariates, etc).

• Open source: we want to use all resources, software and tools that are open source to develop the MET.

• Comparative: we want to provide the capability to view multiple data outputs (e.g., predictions of multiple species), either side by side or through toggling between layers, to gain a deeper understanding of model outputs.

• Multiple formats: ability to upload, read and interact with multiple formats (text, tables, rasters, vectors).