WP7: Validation

Objectives
1. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation. The plan will be iteratively refined for each validation round.
2. Definition and set up of the pilots.
3. Validate the services through pilot evaluation.
4. Generate user requirements (round 1 and 2) or roadmap requirements (round 3).

Description of work
The aim of this WP is to assess to what extent the services provided meet the demands posed. This includes their usability, functionality, fit with contemporary pedagogy and their transferability between systems and situations. To be able to asses this, the work package will provide realistic pilots, to validate the offered services in challenging realistic scenarios. The pilots will cover all languages represented in the consortium, i.e., English, Romanian, Bulgarian, French, German and Dutch and will thereby test whether results can be satisfactorily reproduced across languages. However, given the number of services developed, the number of languages and in order to involve at least two domains, we will opt for ‘smart’ validation. The basic idea behind ‘smart’ validation is not to validate each service for each language and for a variety of domains (e.g. 6 services and 7 languages and 2 domains would imply 84 studies for each round) but make a selection in such a way that all languages are involved, that each service will be tested for at least two languages and that the pilots will cover at least two domains. In this way we can balance between a fair amount of validation effort and reliable feedback on our achievements. In our case it implies that the validation of each individual service may be limited to 2 languages (giving a total of at least 12 pilots). Moreover, the pilots will be focused on two domains, i.e. the medical domain and information technology. In each pilot a minimum of at least 25 students will participate. Learners will be recruited from regular students as well as from lifelong learners.
The data collected will be a combination of quantitative data i.e. system loggings and qualitative data such as ratings of students (and staff to compare their validity) and data resulting from questionnaires, interviews and alike.
The focus of the validation in the first two rounds will be to supply detailed information on usability, effectiveness and efficiency. The WP is closely connected to WP3 for the scenarios and WP2 for the services. The result will be used to improve the services and to derive –if necessary- additional user requirements. The result of the third round will be used as input to a roadmap describing how to sustain the results achieved and a proposal for new directions (cf. WP2).
Participants: OUNL, UU, WUW, UPMF, PUB-NCIT, AURUS KTS, UNIMAN, IPP-BAS, BIT MEDIA