Sales

I-Mass IST-1999-20878 Project I-MASS Work package 2 -Delivery 2.2.1 -Draft version 0.4 Evaluation and Validation Plan

Description
I-Mass IST-1999-20878 Project I-MASS Work package 2 -Delivery 2.2.1 -Draft version 0.4 Evaluation and Validation Plan
Categories
Published
of 18
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  I-Mass WP2 D. 2.2.1 UM/MMI I-MASS frame.rtf Draft 0.5 G. de Haan / F. Pucci Page 1 / 18 24/07/01 IST-1999-20878 Project I-MASS Work package 2 - Delivery 2.2.1 - Draft version 0.4 Evaluation and Validation Plan Geert de Haan and Francesco Pucci with contributions from Johan van de Walle and Gabriël Hopmans Contact: Geert de Haan Maastricht McLuhan Institute P.O. Box 616 6200 MD Maastricht The Netherlands email: G.deHaan@mmi.unimaas.nl tel.: +31-43-3882714  I-Mass WP2 D. 2.2.1 UM/MMI I-MASS frame.rtf Draft 0.5 G. de Haan / F. Pucci Page 2 / 18 24/07/01  Project Number  IST-1999-20878  Project Title  I-MASS  Document Type  PU- Public  Document Number  D2.6.1 (Vs. 0.4) Contractual date of delivery to the Commission  Month 2 of the project  Actual date of delivery to the Commission  14-8-2001 (Draft Version 0.4) Title of the Document  Evaluation and validation Plan Work package contributing to Document  WP 2  Nature of the Document  Report  Author(s): G. de Haan; F. Pucci (contr. G. Hopmans, J. v.d. Walle)  Abstract: Proposal for for the validation framework of I-MASS. There are four areas to validate: metadata mapping mechanisms the multi-agent toolset, the dynamic user models, the virtual reference room user interface. The document will be updated according to the schedule of the project.  Keyword list :   Intelligent agents, Dynamic User Models, I-MASS, IST. User interface, validation, metadata mapping mechanisms,  I-Mass WP2 D. 2.2.1 UM/MMI I-MASS frame.rtf Draft 0.5 G. de Haan / F. Pucci Page 3 / 18 24/07/01 Table of Contents Table of Contents ................................................................................................ 3 1.0 Introduction ................................................................................................... 4 2.0 Summary of the work objectives in the Description of work document .. 4 3 A Proposal for a validation Framework  .......................................................... 5 3.0 Introduction ................................................................................................. 5 3.0.1 Empirical validation and the I-MASS user interface ............................... 5 3.0.2 Empirical and formal validation ............................................................. 6 3.0.3 Scenarios for user requirements and validation ..................................... 6 3.0.4 Using a scenario-based approach in I-MASS  ......................................... 6 3.1 Validating the metadata mapping mechanisms in I-MASS ............................ 7 3.2 Validation issues concerning I-MASS' multi-agent toolset ............................. 7 3.2.1 Validating individual agents ................................................................... 8 3.2.2 Validating a collection of agents ............................................................ 8 3.2.3 Validating the agent-platform used ....................................................... 8 3.3 Dynamic user models for information retrieval in I-MASS .................... 8 3.4 Cognitive Ergonomic issues in the evaluation of the I-MASS user interface ..........................................................................................................10 3.4.1 General usability aspects in I-MASS  .....................................................10 3.4.2 Usability in I-MASS information retrieval and multi-agent system  ........10 3.4.3 Specific usability aspects in I-MASS ......................................................11 3.5 Operational requirements in validating the I-MASS system ................12 4.0 A detailed project plan and time table for validating the I-MASS system .............................................................................................................................12 4.1 Expert-based assessment .......................................................................13 4.1.1 Task-free evaluation method ................................................................13 4.1.2 Task-oriented evaluation methods .......................................................13 4.2 User Assessment ......................................................................................14 4.2.1 Before the Test ....................................................................................15 4.2.2 Scenario-based evaluation: running the test ........................................16 4.2.3 After the test ........................................................................................17 5.0 References ....................................................................................................18  I-Mass WP2 D. 2.2.1 UM/MMI I-MASS frame.rtf Draft 0.5 G. de Haan / F. Pucci Page 4 / 18 24/07/01 1.0 Introduction The purpose of this document is to provide a framework, a project plan, objectives and a time table for the validation in the I-MASS project. In addition, since validation is planned to take place by means of user testing, to relate validation to the user requirements. As agreed among the I-MASS partners, this document will be created in two or three draft versions, one for each new version of the I-MASS system, and one eventually become, in its final version, I-MASS' Validation and Evaluation Report (deliverable D2.6.1). The evolution of this document is related to the following milestones: month 7 - D 7.1.1 prototype MAS I (syntactic interoperability) month 17 - D 8.3.1 validated MAS I month 15 - D 7.4.1 prototype MAS II (semantic interoperability) month 25 - D 8.6.1 validated MAS II month 22 - D 8.7.1 prototype MAS III (user interaction) month 33 - D 7.9.1 validated MAS III This document is planned to evolve as follows:  Version 1 describing the overall plan draft: April 2001 (month 4) final: august 2001 (month 8)  Version 2 describing the plan updated for interoperability draft: march 2002 (month 15) final: September (month 22)  Version 3 describing the plan to test user interaction (MAS III) draft: (month 30) final: October 2003 (month 33) In section 2 of this document we start with a summary analysis of what is said about validation in the Description of Work. In section 3 we attempt to sketch a proposal for the validation framework. Eventually, this section is to contain sections about the four major areas for validation in I-MASS discussing the evaluation of (1) the metadata mapping mechanisms in section 3.1, (2) the multi-agent toolset in section 3.2, (3) the dynamic user models in section 3.3, and (4) the virtual reference room user interface in section 3.4. Section 3 concludes with an operational list of evaluation objectives in section 3.5, which may also be read as a list of research objectives. In section 4 we attempt to sketch a more detailed project plan and time table for validating I-MASS. 2.0 Summary of the work objectives in the Description of work document Objectives as stated: making heterogeneous, large, and distributed data repositories universally accessible  Aiming to solve problems with respect to: - interoperability of content - scalability - user-friendliness/usability By means of: - prototype multi-agent systems - tools - methodologies  I-Mass WP2 D. 2.2.1 UM/MMI I-MASS frame.rtf Draft 0.5 G. de Haan / F. Pucci Page 5 / 18 24/07/01  As a first step towards: - virtual reference rooms The objective requires the following work: - development of mapping and switching mechanisms to facilitate the interoperability between digital, distributed and heterogeneous sets of meta-data - prototype tool sets and technologies to build multi-agent systems to deal with syntactic and semantic interoperability - dynamic user models, which provide users with more responsive software agents The multi-agent models will be tested and validated by user groups (cf. project summary)  Validation is performed by: 1 - developers - technical aspects: 2 - agent specialists - technical aspects and control 3 - expert users - content and the integration thereof 4 - general users - content and the usability aspects In the last two cases, validation is performed with respect to: - syntactic interoperability - semantic interoperability - user responsiveness  Validation of each of the three prototypes is performed twice; once for each test-bed: 1 - after building a prototype to fit one of the test beds 2 - after transferring the prototype to the other test bed 3 A Proposal for a validation Framework 3.0 Introduction On the basis of the information in the Description of Work we distinguish four key issues in the validation of I-MASS: 1 - metadata mapping mechanisms 2 - multi-agent toolset 3 - dynamic user models 4 - the I-MASS user interface 3.0.1 Empirical validation and the I-MASS user interface Evaluation of the user interface is added for the reason that the description of work proposes to validate the I-MASS system by means of user tests. In order for users to access the I-MASS system it is necessary to provide a user interface. Because the usability of the I-MASS system depends on the (dynamic) interplay between the user interface and the functionality, it is not possible to evaluate the first three objectives while not taking the user interface into consideration. In slightly different words, from the perspective of the users the I-MASS system simply is its user interface. To the extend that evaluation does not proceed in an empirical manner, it may very well be possible to distinguish between validating the different parts of the I-MASS system without taking the user interface into consideration but not so when validating the I-MASS system empirically.
Search
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x