In a research and development project such as SAM, it is of course very important to determine to what extend the output created meets the goals and expectations set out at the beginning of the project. In the case of the SAM project, the user engagement and satisfaction with the platform that has been created during the course of the project is pivotal for this. Hence, evaluation activities involving end users play a key part in the overall evaluation of the SAM Platform and the SAM project as a whole.

End users completing evaluation questionnaires for SAM

End users completing evaluation questionnaires for SAM

User Evaluations in the SAM Project

User evaluations in SAM are carried out in two rounds: in a first round, a formative assessment of a functionally complete but not yet final version of the SAM Platform was undertaken in order to determine to gather an initial assessment of how users view the SAM Platform and where the focus for final improvements should be placed. In a second round, a summative assessment of the final system prototype is undertaken which is an important input in order to determine how well the project has achieved the goals set out at the beginning.

SAM provides a complete platform both for users in the media industry and for end users of 2nd Screen experiences created with SAM. Hence, two evaluation streams are part of the project.

The SAM backend systems that are used to create and manage 2nd Screen experiences. Evaluations are carried out with representative business users such as content editors. Users undertake one-on-one evaluation sessions both for formative and summative evaluations, in which they use the SAM backend system in order to manage the content they would like to provide or use and create 2nd Screen experiences.

The 2nd Screen experiences that are created using the SAM backend are evaluated with groups of teenage (15-17) school children in a cooperation with two Spanish schools, Colegio la Encaración and Colegio La Milagrosa near Valencia, Spain. In these trials, the participants are invited to use 2nd Screen experiences created and delivered using the SAM Platform and prototype end user app, and to give feedback on the experiences provided.

Results of the First Round of SAM Evaluations

Formative evaluations both from the business user and from the end user perspective have been completed in the first half of 2016. While the summative evaluations from the business user perspective are still going on at the time of writing and the summative end user evaluations take place after the publication of this article, we can already provide first results concerning the overall SAM concept and the overall user experience, in particular from an end user perspective.

The SAM Linker is used to compose 2nd Screen experiences and was evaluated from the business user perspective

The SAM Linker is used to compose 2nd Screen experiences and was evaluated from the business user perspective

The formative evaluation of the SAM Platform backend primarily resulted in a number of concrete recommendations with the key messages being:

1. The 2nd Screen experience being created and the assets in the system must be presented to the backend users in the same way they will be shown to end users;

2. The system should clearly guide users through workflows instead of grouping by functionalities only, so that for instance the steps for creating a 2nd Screen experience are clear to the business user.

The system reviewers in the formative evaluation appreciated the wealth of functionalities provided by the SAM Platform and recommended to focus on making them more accessible and easier to cross-reference with the experience that is being created during the experience creation process.

The end user evaluation was carried out at the two partner school location over two days and involved just below 60 participants overall. Participants were asked to carry out tasks to test functionalities of the SAM Platform and to complete questionnaires. They were also asked to provide feedback on the SAM end user smartphone/tablet app that they were invited to use.

The dynamic community chat function shown here was evaluated with school end users

The dynamic community chat function shown here was evaluated with school end users

Selected results of the final questionnaire that participants were asked to fill in at the end of the evaluation sessions highlight what we identified as the key outcomes for improving user acceptance of the final prototype. The two figures below illustrate the user responses to questions concerning whether they find two key aspects of the SAM app, provision of additional information and social interaction, useful.

SAM User Evaluation 4SAM User Evaluation 5

 

 

 

 

The responses show that these functionalities are generally considered to be useful by the participants. Our conclusion from this is that the overall concept for the SAM app is sound.

Two additional questionnaire items illustrate where the SAM app has more room for improvement:

SAM User Evaluation 6

SAM User Evaluation 7

 

 

 

 

These two questions examine ease of use and the attitude towards the SAM app expressed by the participants in the first evaluation. While the responses are overall positive, a larger proportion of participants does not agree with the statements concerning ease of use and positive attitude towards the SAM app.

As a result of this, the consortium has initiated steps to improve the ease of use and the appeal of the SAM app for the final end user evaluation of the SAM Platform. The improved final version of the SAM app will be used for final user evaluations, on which we will report in the deliverable document D8.4.2 at the end of the project. After the project has ended, this document will be made available here.

We hope that you have found this short summary of our evaluation interesting. Please contact us if you have any further questions.

Also read about:

Spoken Dialogue Capabilities in SAM

Socialising Around Media with Wi-Fi Aware™

The Specialization of Communication

How SAM provides an endless stream of relevant & rich content

Surfing an Ocean of Extended Content

Adaptation in Spoken Dialogue for Second Screen Interaction

and: SAM and Marvelous Metadata

For additional information about the technologies involved in the SAM project take a look at our public wiki page and/or  follow us via our Twitter channel.