quick link to WP3 FLR

With respect to evaluating the technical performance and the effectiveness of the operational management evaluation framework


The overall approach uses simulation models which are based on stochastic simulation techniques, which take account of uncertainties (parametric as well as structural uncertainty) as well as include sensitivity and risk analysis and assessment. The established management evaluation framework considers uncertainties in the dynamics of the case stocks and fisheries, and emphasis is placed on many kinds of uncertainties including those found in the data collection (calculating variance in input data), assessment, modelling, advisory, management and the implementation processes. By being capable of evaluating the relative performance of multiple alternative management options the evaluation framework has strong capacity in performing sensitivity and risk analyses of the consequences of the various options. This is done by running different scenarios with changed input values based on the scenario analysis and evaluation methodology. The trade-offs implied by various strategies and options can be evaluated through scenario analysis, i.e. be evaluated for their robustness to uncertainty before implementation. As such the facilities and capabilities for the technical evaluation of the established evaluation framework is implicitly build into in the developed framework.

Introduction to Approach

Fisheries management is basically about balancing conflicting objectives. Evaluating the trade-off between different objectives with different time horizons is therefore central to management decisions, and the established management evaluation framework should be able to inform the management decision process in this respect. However, informative comparison of outcomes for different objectives will require that the framework is able to analyse disparate and alternative types of outcome, and that results can be communicated.

The aim of the project is to guide fisheries managers and different stakeholders, including the catching sector, in their capability to make strategic choices. Hence, the project aim to provide outputs that enable to compare alternative options, e.g. in terms of stocks and economic returns for fleets and the industry, but it is not aimed at providing absolute performance measures and predictions for each option to address e.g. the questions: which strategy is likely to give better returns relative to objectives than another strategy? The management evaluation framework can answer “What if ? Questions” like if one fishery is reduced compared to another fishery or if the growth dynamics in a stock changes from a to b - what then? However, given the complexity of the ecosystems and fisheries systems evaluated the framework can not include all dynamics in the system, and as such the predictive power is limited. It is difficult to determine the prediction power of the framework and this will always be a case specific consideration. The output of the evaluation framework is in the best case of the nature: “It is likely that management regime A gives a better performance than management regime B” with respect to a selected “Measure of Performance.” It is in general necessary to be careful with using the models and tools in the evaluation framework as robust predictive tools given the complexity of the systems they are trying to comprehend, and the output from the models should not be used to absolutely quantify complex scenarios un-critically.

The objectives and contents of this technical evaluation report is to give an overview description of the technical facilities, capabilities and utilities of the technical evaluation facets in the framework including uncertainty and error assessment, sensitivity analysis, and risk assessment of the established fisheries management evaluation framework as well as its predictive power.

The technical evaluation is a part of the overall evaluation of the established management evaluation framework where the “Overall Evaluation” include both the “Process Evaluation” and the “Technical Evaluation.”

The “Process Evaluation” is described in other reports including the EFIMAS Conference Summary Report as well as the Stakeholder Focus Group and Interview Reports produced under the EFIMAS Project (WP5 in cyclic feed-back with WP3-WP4). The process evaluation focus on participatory management and evaluation through cyclic feed-back from stakeholders in relation to output and results produced by simulations and modelling under the management evaluation framework when running different scenarios of management as well as running different options for input to the scenario-evaluation which can be changed by the stakeholders.

The “Overall Evaluation” includes testing the general utility of the developed operational management evaluation framework through:

Iterative and cyclic process between WP3, WP4 and WP5 including feedback from regional workshops

Evaluation of the efficiency of the evaluation framework to capture changes in the fisheries systems

Applicability in other stocks / fisheries (general utility evaluation)

A part of the “Overall Evaluation” covers the technical evaluation, where there are implicit and build-in-facilities, capabilities and utilities of the established management evaluation framework(s) to perform the technical evaluation which among other include the following:

Technical Tool Evaluation

  • Evaluation of uncertainty and errors
  • Evaluations carried out on sensitivity, robustness, predictive power and limitations in use and set-up of the models and tools in the framework

- Sensitivity analyses

- Evaluation of robustness

- Evaluation of predictive power

- Evaluation of limitations in use and set-up

  • The codes of evaluation tools are proof-read and tested by alternative coding in critical cases
  • Real and simulated data sets are compiled representing a wide range of data properties, system characteristics and different hypothesis about the underlying processes
  • Sensitivity tests are performed and the robustness of the evaluation framework and descriptive models are evaluated by using them on basis of these diverse data sets

The technical evaluation include rigorous tests of the technical validity of the model implementa¬tion, sensitivity tests and the robustness in relation to data error and to assumptions about the resource system, the fisheries and management implementation. The technical evaluation also evaluates the utility in terms of the technical requirements for set-up and use of the evaluation framework which needs to be done on case specific basis. Examples of the technical evaluation are given on case specific basis and presented under the individual case studies (see EFIMAS DocuWiki as well as the EFIMAS WP4 Technical Reports)..

The Technical Evaluation is described in a special section under WP3 at the EFIMAS DocuWiki as well as under the section WP 5.1 at the DocuWiki in respect of this report, as well as in the Technical Reports associated to WP3 (e.g. Deliverable 3.4, Report of Final Software Package with Documentation). Consequently, this constitutes the general structure, content and ideology of the technical evaluation capability and facilities of the established generic management evaluation framework under the EFIMAS project. Furthermore, the technical evaluation is described in detail on case specific basis in the technical evaluation associated to each case study in Deliverable 4.2, 3.4 and at the EFIMAS DocuWiki under WP4. As such Deliverable 5.2 Technical Evaluation Reports is included in the above reporting and deliverables also. Deliverable 5.2 is reflecting task 5.1, The Technical/Parametric Evaluation, which is covered under Deliverable 3.3, 3.4 and 4.2 (by project month 48) with respect to describing how to evaluate uncertainty, sensitivity, robustness, and to perform risk analysis within the established management evaluation framework(s). Besides this summary EFIMAS has in cooperation with other projects produced additional technical reporting in two more detailed descriptions of the technical evaluation facilities of the established framework(s).

Thus, more details and facets of the technical evaluation facilities of the established framework(s) are comprehensively described in the present report:

  • EFIMAS 2008. Technical Evaluation Summary Report. EFIMAS EU SSP8-CT-2003-502516 Project Report, September 2008. DTU Aqua, DK.

Technical Evaluation Summary Report

associated with the two following reports:

  • Mosqueira, I. 2008. Validation, Verifacation and Testing of software and models in FLR. Report CEFAS, UK, September 2008: 6 pp.

FLR Validation, verification and testing

  • Sparre, P. J. 2008a. User’s Manual for the EXCEL Application “TEMAS” or “Evaluation Frame”. DTU-Aqua Report 190-08: 182 pp. ISBN 978-87-7481-077-3.

Users's Manual for TEMAS Evaluation Frame

as well as through the WP3 Technical Report and the case specific technical evaluation through case studies in the WP4 Technical Report by Month 48 (as well as at EFIMAS WP3 and WP4 DocuWiki pages).

efimas1/wp3/3-3-5.txt · Last modified: 2008/12/07 21:33 by admin
Except where otherwise noted, content on this wiki is licensed under the following license:CC Attribution-Noncommercial-Share Alike 3.0 Unported
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki