D4.3.2 FHW Experiment Progress Report v1.0

D4.3.
2 Experiment progress report including intermediate results

2012-10-31
Manolis Wallace (FHW), Pavlos Mavridis (FHW), Anthousis Andreadis (FHW)
This deliverable is an intermediate progress report for EXPERIMEDIAs embedded experiment focusing on shared, real-time, immersive and interactive cultural and educational and executed by the Foundation of the Hellenic World at its premises at Hellenic Cosmos in Athens. Starting from the more abstract scenario description provided in the earlier D2.1.2, exploiting the architectural blueprint described in the D2.1.3, taking into consideration the methodological guidelines described in D2.1.1 as well as the ethical oversight principles described in D5.1.1, and of course based on the preliminary work described in D4.3.1, the document provides an overview of the work that has been done to this day, the data gathered, the conclusions drawn and the plans for the future of the experiment.
www.experimedia.eu
EXPERIMEDIA
Dissemination level: PU
Project acronym EXPERIMEDIA Full title Experiments in live social and networked media experiences Grant agreement number 287966 Funding scheme Large-scale Integrating Project (IP) Work programme topic Objective ICT-2011.1.6 Future Internet Research and Experimentation (FIRE) Project start date 2011-10-01 Project duration 36 months Activity 4 Experimentation Workpackage 4.3 EX3: shared, real-time, immersive and interactive cultural and educational experiences Deliverable lead organisation FHW Authors Manolis Wallace (FHW), Pavlos Mavridis (FHW), Anthousis Andreadis (FHW) Reviewers Stephen C. Phillips (ITInnov) Version 1.0 Status Final Dissemination level PU: Public Due date PM14 (2012-11-30) Delivery date 2012-10-31
Copyright FHW and other members of the EXPERIMEDIA consortium 2012
EXPERIMEDIA
Table of Contents
1. 2. 3. Executive Summary............................................................................................................................ 4 Introduction ........................................................................................................................................ 6 Background.......................................................................................................................................... 7 3.1. 3.2. 3.3. Goals and success evaluation criteria ..................................................................................... 7 Experiences offered before EXPERIMEDIA ..................................................................... 8 The experience examined in the embedded experiment ..................................................... 9
3.3.1. First stage of the experiment ............................................................................................... 9 3.3.2. Second stage of the experiment ........................................................................................ 10 3.4. 4. 4.1. 4.2. 4.3. 5. 5.1. 5.2. 5.3. 5.4. 6. 6.1. Constraints ............................................................................................................................... 10 Definition of agents ................................................................................................................ 11 Live streaming experiment during the show ....................................................................... 11 Augmented reality experiment after the show .................................................................... 13 Streaming experimental facility ............................................................................................. 16 Augmented reality experimental facility ............................................................................... 16 Social networks experimental facility ................................................................................... 23 Monitoring experimental facility ........................................................................................... 25 Before the beginning of the experiment .............................................................................. 26 Experimental facility architecture ................................................................................................... 11
Experimental facility implementation............................................................................................ 16
Experiment execution ...................................................................................................................... 26 6.1.1. Deploy the facility ............................................................................................................... 26 6.1.2. Familiarize ourselves........................................................................................................... 29 6.1.3. Inform staff and schedule accordingly ............................................................................ 29 6.1.4. Prepare the informed consent forms ............................................................................... 29 6.1.5. Prepare the questionnaire. ................................................................................................. 30 6.2. Experiment procedure............................................................................................................ 30 6.2.1. Preparatory phase................................................................................................................ 30 6.2.2. Execution phase .................................................................................................................. 30 6.2.3. Data acquisition phase ....................................................................................................... 30
7.
Data analysis ...................................................................................................................................... 32 7.1. 7.2. Statistical views of the data .................................................................................................... 32 Interesting observations ......................................................................................................... 33 2
EXPERIMEDIA 7.3.
Evaluation ................................................................................................................................ 34
7.3.1. Internal evaluation .............................................................................................................. 34 7.3.2. External evaluation ............................................................................................................. 35 8. Current status and future plans ...................................................................................................... 36 8.1. 8.2. 9. 9.1. Experimental facility ............................................................................................................... 36 Experimental methodology ................................................................................................... 36 Minimum ethical principles ................................................................................................... 38
Ethics, privacy, PIA.......................................................................................................................... 38 9.1.1. Doing good .......................................................................................................................... 38 9.1.2. Doing no harm .................................................................................................................... 39 9.1.3. Risk management ................................................................................................................ 39 9.1.4. Consent................................................................................................................................. 39 9.1.5. Confidentiality ..................................................................................................................... 40 9.1.6. Data protection ................................................................................................................... 40 9.2. Ethical oversight principles ................................................................................................... 40 9.2.1. Informed consent ............................................................................................................... 41 9.2.2. Deception ............................................................................................................................. 41 9.2.3. Data collection..................................................................................................................... 41 9.2.4. Withdrawal from the investigation ................................................................................... 41 9.2.5. Observational research ....................................................................................................... 42 9.2.6. Data protection regulation................................................................................................. 42 9.2.7. Consortium partner responsibility .................................................................................... 42 9.3. 10.1. 10.2. PIA ............................................................................................................................................ 42 Evolution and handling of risks ............................................................................................ 44 Current risk registers............................................................................................................... 45
10. Risks.................................................................................................................................................... 44
10.2.1. Risks for the participants ................................................................................................... 45 10.2.2. Risks for the experiment .................................................................................................... 46 11. Conclusion ......................................................................................................................................... 48 Appendix A. Appendix B. Appendix C. Appendix D. Appendix E. Informed consent ........................................................................................................ 50 Device lease .................................................................................................................. 51 Structured questionnaire ............................................................................................. 52 Gathered data ............................................................................................................... 54 Independent experiment evaluation.......................................................................... 55 3
EXPERIMEDIA
1. Executive Summary
This document presents the working report of the 3rd EXERIMEDIA embedded experiment. The document covers all aspects of the experiment, ranging from the purely technical to the purely theoretical ones. More specifically: Section 2 provides a brief introduction to the document, including a summary of the reasoning that led us to split the experiment in two stages. Section 3 highlights relevant elements of the experiment's background in order to improve the readability and completeness of this document, whilst the reader is advised to refer to previous deliverables of the project for further information. Section 4 makes the architecture descriptions of D4.3.1 more detailed and adopts them to the specifics of the first stage of the experiment. The architecture is presented for the two distinct parts of the experimental facility and the information flow between the different components is also shown. Section 5 continues on the same path and explains how these architectures where actually implemented. Moving on to the actual implementation of the experiment, Section 5 describes how the experimental facility was implemented. It is worth noting that due to the early timing of our experiment additional work had to be carried out from our part in order to address issues not yet addressed by our technical partners; occasionally not even successfully. Section 6 focuses on the actual execution of the experiment and gathering of the data. Using the implemented experimental infrastructure and following a reduced version of the scenario already outlined in D4.3.1 we explain how we invited a number of participants to experience the EXPERIMEDIA extensions to the venue and provide us with their feedback on them. Unfortunately we were only able to do this for the augmented reality component at this stage, but we are confident we will also cover the other components for the second stage of the experiment. Section 7 presents the gathered data and based on a brief statistical processing reaches some initial conclusions. Despite the small sample they are based on, these conclusions will be very useful in guiding the preparations for the second stage of the experiment. Evaluations of our work, both internal and independent, are also presented. Having presented the work performed to this day, in Section 8 we also outline the work planned for the second stage of the experiment, so that all of its goals may be reached. Section 9 discusses ethics and privacy. The precautions taken with respect to ethics and privacy were already analysed quite rigorously in D4.3.1. We do revisit the subject here for completeness but the reader should be made aware that much of the text of this section is in fact also found in section 6 of D4.3.1, but here it is adapted to the specifics of the first stage of the experiment, as they finally formed.
EXPERIMEDIA
Risks have been monitored and handled throughout the design, implementation and execution of the experiment, following the guidelines specified in D1.1.2 and using the registers defined in D4.3.1. A complete review of the evolution of the risk registers is provided in Section 10. Appendices include the form used to get informed consent, the form used to track devices, the questionnaire used during the experiment, and the complete listing of the data gathered. We also include a copy of the external evaluators report.
EXPERIMEDIA
2. Introduction
This deliverable is an intermediate progress report for EXPERIMEDIAs embedded experiment focusing on shared, real-time, immersive and interactive cultural and educational and executed by the Foundation of the Hellenic World at its premises at Hellenic Cosmos in Athens. Starting from the more abstract scenario description provided in the earlier D2.1.2, exploiting the architectural blueprint described in the D2.1.3, taking into consideration the methodological guidelines described in D2.1.1 as well as the ethical oversight principles described in D5.1.1, and of course based on the preliminary work described in D4.3.1, the document provides an overview of the work that has been done to this day, the data gathered, the conclusions drawn as the plans for the future of the experiment. Very early in our work towards the execution of this experiment we came to realize some difficulties with the scheduling of different tasks and events in the timeline of the project that could prove problematic. Most notable among them the fact that the first working versions of the technical partners contributions were expected late in the project and in fact after the finalization of the plans for the experiment as well as the fact that there would not be a chance for us to have feedback from the project reviewers at a mature stage of the work in order to be sure that we are headed in the right direction. Both constitute very serious dangers for the experiment and therefore appropriate action was required. Therefore, in order to navigate away from the aforementioned dangers we decided to split the experiment in two stages with very distinct character and goals. The first stage is aimed to run quickly, provide an early confirmation of the projects methodology and technical approach, put technical components to a practical test and generate feedback for the technical partners who are working on them, gather know-how that will help the new partners who are just joining the project to seek even higher goals, allow the consortium to have a demonstrable output from early on in the project and give us an opportunity to present our approach to experimenting to the project reviewers during the first year review. The second stage is aimed to build on the experiences and know-how of the first stage in order to perform the complete experimental work that was envisaged for the 3 rd embedded experiment in the most suitable manner possible. This document reports on the first stage of the experiment, which has just been completed. The document covers all aspects of the experiment, ranging from the purely technical to the purely theoretical ones, and also hints at the work planned towards the successful implementation and execution of the second stage of the experiment as well.
EXPERIMEDIA
3. Background
The experiment's background has already been discussed in D4.3.1 as well as in D31.1, D31.2 and D4.33. We briefly review here some main elements of it in order to enhance the readability and completeness of the current text and advise the reader to refer to the abovementioned documents for further details.
3.1.
Goals and success evaluation criteria
The embedded experiment involves the development of an experimental facility and its testing with the participation of real users in real settings. Clearly something will be implemented, some data will be gathered and some analysis will happen. Still, a core question remains unanswered and critically subjective: when has the experimenter done enough? In order to remove the subjectivity and provide a clear measure of success, we identified in D4.3.1 a set of goals for the experiment and defined the corresponding objective criteria. Given the fact that four different levels of success are defined, the objective is to achieve a considerable level of success in the first stage of the experiment reported herein and then build on that in the second and final stage of the experiment in pursuit of even higher success. We review the experiment's goals here briefly, together with their corresponding success criteria, as defined in D4.3.1: Goal 1: Be an EXPERIMEDIA test bed (Baseline success) The experiment can be executed. This entails having implemented the experiment architecture, having made all of the included components operational and having been successful in their integration. Know-how has been gathered. This refers to the gathering of know-how related to the further implementation of the embedded experiment.
Goal 2: Explore suitability of FIRE technologies for the field under examination (Moderate success) Identify differentiation between using and not using the FIRE technologies. In other words, we need to establish that there is a substantial difference for the visitors between the conventional experience currently offered and the one that will be offered in the scope of the experiment. Classify the impact of each component as positive, negative or neutral. This is a more specific version of the previous criterion, as here it is not enough to establish that there is a difference. What is also required is a clear indication regarding whether this difference has an impact that QoE of the visitors and if so whether this impact is positive or negative.
Goal 3: Measure impact of FIRE technologies (Success)
EXPERIMEDIA
Quantify and measure QoE. In other words we need to have designed a measure that quantifies QoE and we also need to have applied this to data gathered from the experiment. Correlate measured QoE to utilized FIRE technologies. Moving a step further, to meet this criterion we should be able to identify the contribution of each component in the QoE, so that strategic decisions can be made regarding the directions that warrant further examination.
Goal 4: Identify parameters that affect impact (Exceptional success) Measure QoE for different parameters. This criterion is met if data gathered when running different instances of the experiment and different QoE values are computed. The compared instances need to be such that a direct comparison related the differences in QoE to differences in QoS of some kind (e.g. bitrate), differences in the design and execution of experiment (e.g. duration of show), differences in demographics etc. Gathered insight for the design of future experiments. This refers to the gathering of insight related to the implementation of future EXPERIMEDIA installations at Hellenic Cosmos, for example in order to run future experiments.
3.2.
Experiences offered before EXPERIMEDIA
The embedded experiment is built around the VR immersion experience offered by the Tholos. When EXPERIMEDIA extensions are not considered, this is offered mainly as a standalone experience that is not combined with any of the other exhibits or services of Hellenic Cosmos. The typical operation of the Tholos and of the service it offers to its visitors may be graphically modelled as in Figure 1. It is easy to see that this is a mainly one-way communication system, as the museum educator controls the system, thus specifying what the Tholos system will render and project to the visitors, while at the same time commenting on it. As a sole exception to this, visitors are able to participate in electronic polls which determine the path that the Educator will follow altering in this way the flow of the presentation in real time. The main reason for this extremely structured and predefined approach is that the museum educator is working with predefined scenarios, i.e. descriptive texts prepared by the FHW experts. These texts provide information on the 3D worlds in a specific order and therefore the tour in the 3D world has to follow the same order, otherwise the museum educator would be unable to provide synchronized information. Before the start of the show there is a brief pre-show which informs the visitors about the characteristics of the VR immersion system. This contains information such as whether they will feel dizzy, what to do in that case and so on. The pre-show is not related to the content of the show and does not add to the value that is offered to the visitors. After the end of the show the visitors exit the Tholos and this completes the offered experience.
EXPERIMEDIA
Figure 1. Tholos operation without EXPERIMEDIA extensions.
3.3.
The experience examined in the embedded experiment
As we have already mentioned in our introductory comments, we have had compelling motivation to run the experiment with an accelerated schedule in time for it to be examined during the projects first yearly review. On the other hand, the projects very tight time schedule made it almost impossible for the complete experimental facility to be implemented and used and then for all this date to be analysed in time. Particularly, the timing of the delivery of the first working Act2 software as well as of the Act3 integration work made it next to impossible to run an experiment within the first year. In order to meet our goal of running and reporting the experiment for the first review we were forced to create a scaled down version of it, one containing only the components that were available ahead of their planned time, so that the reviewers feedback on our direction could be sought. As a result, our experiment is now organized in two stages, as explained in the following.
3.3.1. First stage of the experiment

As will be explained in more detail in the following sections, for the first stage of the experiment we have only implemented the streaming and the augmented reality parts. Moreover, due to scheduling constraints, we have only executed and analysed the augmented reality part.
EXPERIMEDIA
3.3.2. Second stage of the experiment

For the second stage of the experiment we shall implement and test the whole experimental facility as planned. The only note is that based on the evolution of the risks listed in section 10 there may arise the need for slight deviations from the plans presented in deliverable D4.3.1. As the plans have now, to move to the second stage of the experiment we will also have to integrate the experiment monitoring component as well as the social component in the current experimental facility and we will have to run the experiments with more people and analyse the data.
3.4.
Constraints
The main attendants of FHW shows are children and adolescents. In the EXPERIMEDIA experiments only adults will be considered, which creates a question regarding the validity and generality of the results. For the same reasons, and given the fact that the navigation in a virtual world is a group experience, it is not possible to apply some monitoring techniques (for example video recording), unless if the experiment is executed only when pure groups of participants are present, i.e. when there is no one present who is either not eligible or has not agreed to participate in the experiment. Regarding the experiment timeline, since some of the real exhibits connected to tags in the virtual content are in open areas of the Hellenic Cosmos venue, the weather may have an impact on the execution of the experiment. The execution of the experiment is also constrained by the timeline of the development of EXPERIMEDIA components by the technological partners of the project. This has already become more than evident during the definition and implementation of the first stage of the experiment.
10
EXPERIMEDIA
4. Experimental facility architecture

As has already been made clear in D4.3.1, the embedded experiment had and still has a very tight schedule. As a result, and for reasons that will be discussed in further detail in section 10, minor adjustments to the original plans had to be made during implementation and execution, as to make sure that the experiment remained both on track and on schedule. Therefore, before moving on with the detailed description of the implementation work that has been carried out since the submission of D4.3.1, we review in the following the design for the implementation of the experimental facility and provide its most current instance. Specifically, we discuss the parts of the experimental facility and their architecture. Our experiment consists of two parts. The first part takes place during the show, inside the dome installation of the Foundation of the Hellenic World, and aims to improve the overall experience of the audience through their interaction with scientists and experts. The second part of the experiment takes place after the show, and uses augmented reality technologies to provide additional information to the audience. Below we provide an architectural overview of each part.
4.1.
Definition of agents
The agents participating in this experiment are - Visitors: The audience that participate in the interactive shows of the "Tholos" dome theater. We will refer to them as Visitors. - Museum educator or simply "educator" is a person that interacts with the audience and controls the navigation through the virtual 3D environment. - Experts: One or more scientists, located remotely, that provide additional commentary on the content that is shown during the show and answer questions from the visitors. We will refer to these scientists as "Experts".
4.2.
Live streaming experiment during the show
Since the Experts are on a remote location, the main motivation behind this experiment is to allow real-time interaction between the Experts and the Visitors. To this end, the experts and the audience must see and hear the same part of the interactive show what is shown in the dome. Additionally, the experts must be able to hear any questions from the audience, and the audience must hear the experts. Figure 2 shows a high level overview of the locations and the desired flow of information between the agents that participate in this experiment.
11
EXPERIMEDIA
Remote Location 1
"Tholos" Theater
Remote Location 2
Expert 1 Audio + Video from "Tholos"
Visitors
Expert 2
Educator
Audio + Video from "Tholos"
Audio + Video from Experts
Figure 2. High-level overview of the locations and the desired information flow between the agents of the experiment. In this particular example two experts participate in the experiment from remote locations.
The actual connectivity and communication between the components is shown in Figure 3. In this figure we can see that the museum educator holds the navigation control, which specifies the content that should be displayed to the visitors. Based on this input, the cluster in the Tholos dome processes the loaded 3D world in order to render the according location and viewpoint and display it to the visitors in the dome. This is the part that was already supported before EXPERIMEDIA and it is in fact the typical scenario for the utilization of the Tholos. With the EXPERIMEDIA extensions, the Tholos system, in addition to the local projections, also forwards the rendered stream (actually a downsized and 2D version of it) to the video stream server, which in turn makes it available to the experts' application. In this way the experts will be aware of the presented content in real-time. The video stream from Tholos is captured from another PC through a video capture card (AVERMEDIA Game Broadcaster HD). At this PC the video is transcoded along with the audio feed from the educators microphone and are transmitted using Adobes Flash Live Encoding to ATOS Server. The Experts will be using ATOS Flash Player to visualize the video stream. Additionally, the museum educator is able to see video-feeds from the experts, as shown in Figure 11. This video and audio feed from the experts is also passing through the ATOS Server, where it is transcoded, and it is made accessible to the educator through a regular website. In order to see this website, the adobe flash player is required, a plugin that is available in most modern operating systems. A minor limitation of the system is that since adobe flash is required, the web-based application cannot run on mobile or embedded devices, which usually lack support for this plugin. Nevertheless, the usage of web-technologies like flash in our system makes the video and audio streams easily accessible from multiple computers.
12
EXPERIMEDIA
4.3.
Augmented reality experiment after the show
The second part of the experiment takes part after the interactive show in the Tholos Theatre. During this experiment, the visitors can learn additional information about the content of the show, by visiting specific locations on the premises of the Hellenic World facilities. In these locations we have installed markers and using augmented reality techniques, these markers can be recognized by an application that runs on commodity mobile devices. The application can the super-impose virtual objects on top of the real ones, by tracking the position and orientation of the markers, or can simply open a website with additional information, depending on the location of the marker. It is worth noting that this experiment does not require the participation of the experts, the visitors will take the additional information directly from their mobile device. This is an significant advantage, because it minimizes the operating expenses. The architecture and information flow of this part of the experiment is shown in Figure 4. With our application, the input video feed from the real world is augmented with virtual 3D objects. Alternatively, when a specific marker is detected in the input video feed, the user is redirected automatically to a website that is related to his particular location. The association of the markers with the virtual objects or the websites is performed using a configuration file.
13
EXPERIMEDIA
Figure 3. Flow of information and component diagram for the experiment.
14
EXPERIMEDIA
Figure 4. Flow of information and component diagram for the experiment. Our application configuration can support up to 50 markers and associate them with different 3D objects or links to websites.
15
EXPERIMEDIA
5. Experimental facility implementation

Bellow we outline the implementation efforts that were required in order to support this experiment.
5.1.
Streaming experimental facility
The implementation of the first part of the experiment requires streaming the content shown at the dome to multiple remote locations. The interactive content for the dome is produced by a cluster of PCs, as shown in Figure 2. Each PC renders of this cluster renders only one part of the dome. We have augmented this cluster with one additional PC that renders one extra view of the dome. This required a change in the configuration files of our cluster. The video output of this PC is directed to another PC that encodes the content in MPEG4 H.264 format and streams it to the ATOS central server. The h.264 format was chosen among many others because it is one of the most advanced video codecs and it is known to provide state-of-the-art encoding quality. The quality of the streaming experience is highly sensitive to the available bandwidth. Therefore we have experimented with various encoding settings for the video and audio stream, in order to maximize the image quality and the responsiveness of the streaming content. One technical limitation that was imposed from the ATOS media server was that the stream should be at 60fps. A 30fps stream would be perfectly adequate for our purposes and would require less bandwidth but at the time of this experiment we could not use this option. After some experiments with our line, we have settled on a 1440x1080 video resolution for the stream. This is highly depended to the bandwidth of the internet connection in the FHW facilities. It should be noted that the ATOS media server is located in Spain. Therefore some latency is to be expected in the communication. Aside from the network latency, which is to be expected, some amount of latency was also introduced by the h.264 encoder in our streaming PC. Nevertheless, this latency was only a few seconds, and did not pose a serious problem in our experiment. In the future it might be worth investigating methods to reduce this latency even further.
5.2.
Augmented reality experimental facility
The second part of our experiment, as noted before, involves a mobile application that uses augmented reality technologies. The augmented reality framework that was provided to us for testing was the Metaio Mobile SDK. Using this SDK, we have developed our own mobile application. During the implementation of this application we had to make a several design decisions and tests. Bellow we outline these implementation efforts. First we had to decide which mobile platform to support. The provided SDK supports both the Android OS from Google and the Apple iOS. After consideration, we chose to implement our application in the Android OS, because we have found that the development process is more open. Developing and testing on Apple devices required a registration with Apple, something that could take several weeks for corporate entities and could potentially delay our efforts, thus compromise the project. Additionally, the social networking API that is required for another part
16
EXPERIMEDIA
of the experiment supports only the Android OS. Therefore the application was made for Android, using the JAVA language and the Eclipse SDK and the Android Development Toolkit. Figure 5 shows a screenshot of the development environment. The exact versions of the software used are Eclipse SDK 3.7.2, ADK 18.0.0.v201203301601-306762. The target Android OS version was 2.3.7 or higher.
Figure 5. The development environment for the mobile application.
After the decision about the Operating System, we had to choose the particular device that was going to be used in our experiment. We performed our first experiments in a "Sony Ericson Live" mobile phone, shown in Figure 6.
17
EXPERIMEDIA
Figure 6. Initial tests were performed in this mobile phone (Sony Ericsson Live). The small screen made it difficult to use our application.
Our application was successfully running in this phone, but we have observed that the screen size was rather small, 3.2inches. The small size of the screen was making it hard for the visitors to see the provided information in the form of 3D Objects or Websites. The details in the 3D objects were not apparent and the websites were difficult to navigate and read. Therefore we have used a device with larger screen, the Sony XPERIA P, shown in Figure 7. This phone has a 4.0Inch screen, which makes using the application more comfortable.
Figure 7. The actual device that was used in the experiment (Sony XPERIA P). The 4Inch screen is a good trade-off between usability and portability.
Ideally, we could also use some Tablet PCs running the Android OS, but the larger size makes these devices less portable. We believe a screen around 4-5 inches provides the right balance between usability and portability. Portability is required for our experiment, because the
18
EXPERIMEDIA
augmented reality markers are scattered at various locations in the FHW facilities, and the visitors are expected to carry the device with them, while walking around.
Figure 8. Our application running on the Android Emulator on a PC. Although the application can run in the emulated device, the actual experiments and testing cannot be performed in the emulator, because augmented reality applications inherently require the video feed from a mobile device. Thus, all of our testing and development for the augment reality experiment was performed on actual mobile devices.
Next, we had to adapt our virtual reality content to the needs of a mobile device. Our 3D models are mostly designed to be displayed by powerful workstations. These workstations are several times faster than a mobile device, so our content had to be simplifies in order to be usable in a mobile phone. After considerable experimentation, we have found that our particular mobile device can show 3D models with up to 20000 triangles. Another limitation was that the model should have only one texture, instead of multiple layers of textures that we use in our workstations. To overcome this limitation, our artists merged the information from all the layers in a single texture. This texture also has the information about the lighting. This process is called "baking" in technical terms. Therefore, we have concluded that after reducing the polygon count to around 20000 triangles and baking all the information to a single texture, the 3D models were usable in our mobile devices. Next, we have performed various experiments with the type of markers that we were going to use in our experiments. The Metaio SDK supports both "marker-less tracking", where the
19
EXPERIMEDIA
application can track the location and the orientation of an arbitrary image, and tracking with markers, where the application tracks specific markers, as the ones shown in Figure 9.
Figure 9. Example of markers used in the augmented reality application.
In both cases, the 3D model that is superimposed in the video feed should follow the position and the orientation of the image or the marker respectively. Nevertheless, we have found that the tracking with markers instead of images is more robust. In particular, we have found that when using the markers, the tracking is more stable, meaning that the 3D object is actually closely following (tracks) the marker in the video feed, and does not appear to be moving independently, which was the case when using markerless tracking. In particular, the performance of markerless tracking depends on the contents of the actual image that is used for the tracking, something that we have found unacceptable for our application. Furthermore, tracking with markers was less sensitive to lighting conditions, and was working reliably even on environments with relatively low illumination, thus we have used tracking with markers. The markers are integrated on an information sheets that are placed on various locations at the facilities of Hellenic Cosmos. Figure 10 shows an example of such a sheet, along with the application running on the mobile device.
20
EXPERIMEDIA
Figure 10. Left: The augmented reality information sheet. Right: The visitor places the phone over the sheet and observer the 3D object.
Figure 11. The PC screen of the museum educator, communicating with the remotely located Expert.
21
EXPERIMEDIA
Figure 12. Two museum educators guide the audience through a virtual representation of the ancient City of Miletus inside the Dome Installation of FHW. Low light conditions were required for the projectors. One educator controls the navigation with the joystick, while the other one narrates some historical information. On the right side we can see the PC with the video feed from the experts.
Figure 13. Another view of the educators during the experiment.
22
EXPERIMEDIA
5.3.
Social networks experimental facility
Our intention was to use social networks to allow direct communication of the Visitors with the Experts. To this end, we have used the Social Networking API, which is provided by our partners, in order to build a mobile application that connects the users to the Twitter and Facebook mobile networks. During our testing, we have found that the Social Networking API is not robust and ready for deployment, and further development is needed in order to fix various issues and bugs. Bellow we have a comprehensive description of our tests, the errors occurred and how to reproduce them: 1. The provided source at the Experimedia SVN has a small error and fails compilation. In particular at the link path there are two versions of the twitter4j library. To make it compile and perform the tests that follow we had to remove the "twitter4j-core-2.1.1SNAPSHOT.jar" library and keep only the "twitter4j-core-android-2.2.5.jar". 2. After the above correction, I have launched the example app and tried to login to twitter. When pressing the button I insert my twitter account credentials and I authorize the app. After this step, which is performed in a browser that pops-out, the control redirected back to the android app, but it crashes immediately and shows the typical crash message. See the screenshot in Figure 14.
23
EXPERIMEDIA
Figure 14. An error during the testing of the Social Networking API in the Android Emulator. The same error also occurs on the devices.
3. After this step, if you relaunch the app, it appears that the twitter account is logged-in. Perhaps it kept the credentials from the previous step, before the crash. But if I sent a message, the app crashes again. 4. I have also tried to login only using Facebook. In the Facebook case the login happens again in a browser window that pops-out, but there is not any crash, like twitter. After the login, when trying to send a test message, the app crashes again. Interestingly enough, judging from the logs, the app crashes on the twitter components (TwitterBaseImpl.java). I would expect if the user provided only a Facebook login, and not a Twitter one, the API should not try to use Twitter (and crash) 5. The last test was to try to send a message using both Twitter and FB (it could be an API limitation that it needs both). This use-case also leads to a crash. It seems that the error has again to do with the twitter integration/components of the API. This has been tested on a Sony Experia P with android 2.3.7, on a virtual device with android 2.3.3 and on a virtual device me 4.0.3. All the above configurations exhibited the same behaviour. All the errors appear deterministic and reproducible in our setup (i.e. if we follow the exact same steps, we always get the errors described above)
24
EXPERIMEDIA
More recently we have received an updated version of the Social Networking API from our partners, and we are in the process of integrating it in our experimental facility so that we can include it in our tests, as originally planned.
5.4.
Monitoring experimental facility
To this day we have not worked with any other EXPERIMEDIA components. Clearly, this will be altered for the second stage of the experiment. If nothing else, we shall at least have to integrate the experiment monitoring component that is being prepared by our partners, since the projects methodology requires the experiments to be built and executed around it.
25
EXPERIMEDIA
6. Experiment execution
In this section we present the actions taken and procedures followed in order to conduct the experiment.
6.1.
Before the beginning of the experiment
A great deal of effort was required before even approaching the first candidate participant. Specifically, we needed to: deploy the experimental facility, familiarize ourselves with the experimental facility, so that we could provide assistance as required, inform staff about the upcoming experiment and schedule accordingly, prepare the informed consent forms and, prepare the questionnaire.
6.1.1. Deploy the facility

This refers to the actual selection and development of the interest points for which AR services will be offered. With the help of FHW historians and archaeologists 5 points were identified that are related to Miletus (which is the topic of the Tholos projection that will be used in the context of EXPERIMEDIA) and for which additional content is available. For 4 points a pure AR service is provided as a 3D object will be superimposed on their camera input, whilst for the 5th point there will be a redirect to an online link. Specifically: Point 1 is a CD on Miletus offered in the Hellenic Cosmos store and is coupled with a 3D reconstruction of a bed that could be found in the city.
Figure 15. Point 1
Point 2 is a Book on Priene offered in the Hellenic Cosmos store and is coupled with a 3D reconstruction of a building that could be found in the city.
26
EXPERIMEDIA
Figure 16. Point 2
Point 3 is a presentation of Thales in the Mathematics exhibition. It is coupled with a link to the online Encyclopedia of the Hellenic World (www.ehw.gr) and specifically to the lemma on Thales. As we found out during testing the wireless signal at that location is not stable enough and at times it is not possible to access the page. In face the wireless connection failed us during all the tests and therefore we did not manage to have feedback on this point of interest.
Figure 17. Point 3
Point 4 is a physical reconstruction of an ancient ship. It is coupled with a digital reconstruction of a different ancient ship and the descriptive text outlines the differences. Depending on the time of day and most importantly on whether the weather was cloudy or not, difficulties were observed with the clarity of the monitor.
27
EXPERIMEDIA
Figure 18. Point 4
Point 5 is a sample excavation site that we use to show pupils how an archaeologist actually works. It is coupled with the reconstruction of an amphora.
Figure 19. Point 5
One core concern is the relative location of these points. As can be seen in Figure 20 the points are located at quite distant locations from each other. In fact the walking distance between points 3 and 4 is approximately 900 meters. Considering that a participant would have to be approached and informed about the project and the experiment at the main entrance of the venue (close to points 1 and 2) but should walk to points 4 and 5 and then back again in order to have experienced all 5 elements that we have developed, it is clear that the experiment will take long to run. In fact, we have had to accept that only one round of experiments will run in a day and that in the participant selection criteria we would also add an evaluation of the physical state of the individual, so that they would not have a problem covering this distance.
28
EXPERIMEDIA
5.Amphorea 4. Ship
2. Building
1. Bed
3. Link about Thales
Figure 20. Venue outline
6.1.2. Familiarize ourselves

This was the easy part, at lease for the first stage of the experiment. For the reasons that are presented below we were forced to only run the part of the experiment with the augmented reality component. The related application is quite easy to use (once it is turned on all a user has to do is point to the card) so not much training was needed for Hellenic Cosmos personnel. For the second stage of the experiment we expect to have a considerably harder task in providing both the museum educators and the experts with effective operational guidelines.
6.1.3. Inform staff and schedule accordingly

When it comes to the augmented reality component no scheduling was necessary. For the Tholos, on the other hand, we needed to find free time in the Tholos schedule, book it for EXPERIMEDIA, arrange with ATOS so that streaming would be available, arrange with the museum educators and make sure the experts were also available a the same time. In trying to arrange for this we came to realize that the expert has to be a person that knows the content much better than the museum educators (who are also historians and archaeologists) for this to make sense. There are only two such individuals in FHW (they lead our archaeological research) and as it should be expected their schedule is very full and their time very limited due to the fact that they hold critical roles in multiple projects. Unfortunately we did not manage to find a time slot that could accommodate all the above constraints and had to leave this part of the experiment for the second stage, so that we have more time to either re-examine the scenario or book ahead.
6.1.4. Prepare the informed consent forms

The informed consent form was prepared based on a sample that was provided by the EAB. It can be found (in Greek) in Appendix A. 29
EXPERIMEDIA
6.1.5. Prepare the questionnaire.

The questionnaire was prepared based on the ethical and privacy guidelines that were agreed upon. Therefore only multiple choice questions are included. The questions examine the participants views regarding the clarity of the image, the relevance of the content to the point of activation, the participants interest in the presented content, the added educational value and the fun they had using the application. It also asks whether the participants would accept to pay a fee of 1 euro to have access to this application, or to similar application with more points of interest. All the above are asked both generally and also separately for each point of interest. The questionnaire can be found (in Greek) in Appendix C
6.2.
Experiment procedure
6.2.1. Preparatory phase

In this stage we selected participants. This was generally done early in the day so that enough time would be available for the participants to stroll to all points considered in the experiment. In fact candidate participants were asked beforehand whether they would be willing to take a stroll to the other end of the venue and then return. As agreed, no minors were approached. The experiments goals, the overall procedure, their role in the experiment, the nature of the gathered data, the handling of the data etc were explained in detail to every candidate aurally, and then the written consent form was presented. We were available to answer any additional questions posed, but there were no such incidents. The experiment run over a duration of 4 days, and since only 3 devices are available this means that we worked with a total of 12 participants.
6.2.2. Execution phase

This is the phase in which the visitors that participate in the experiment visit the Tholos as well as the physical exhibitions of the Hellenic Cosmos and use the augmented reality application. The participants were required to sign for the smart devices. This documentation was not combined with any of the experiments data and was only maintained until the end of the tour, when the device was returned. The form signed for the lease of the smart device is presented in Appendix B. The brochure described in D4.3.1 was not created, as we chose to physically place the markers at the interest points. This meant of course that we had to provide participants with directions regarding the location of the interest points.
6.2.3. Data acquisition phase

Since only the AR part of the experiment was executed, Visitors were the only participants. We used questionnaires and not focus groups to poll them for information. 6.2.3.1. Questionnaires The questionnaire used has already been presented.
30
EXPERIMEDIA
6.2.3.2. Focus groups In the first stage of the experiment it proved impossible to hold focus groups. The reason is purely technical: focus groups work well when a large number of participants has first been polled through structured questionnaires and then based on the analysis of these questionnaires the researchers determine some points on which they would like to have additional information and organize focus groups. Each focus group has a smaller but still considerable number of members and through discussion the point of focus is examined more to get a more concrete idea of the participants view. In the first stage of the experiment it was first of all impossible to first examine the data and then have the focus groups, simply because the data was gathered over different days and when it was analysed the experiment execution was already over and no participants were available to hold a focus group. But even if we had manually selected some focus questions and decided to hold focus groups we still could not do it because: The three participants that we have as a maximum number for any given moment is still a very small number The participants are scattered over a very large area and randomly return to the reception, hand in the equipment, fill in the questionnaires and leave. There is no way to synchronize the departure of the three participants and therefore it is not possible to hold focus groups since it is not possible to have the participants at one place together at the end of the experiment.
For the second stage of the experiment we plan to hold focus groups. The execution of the part of the experiment that refers to Tholos will facilitate this, as considerably larger numbers of individuals can take part at the same time.
31
EXPERIMEDIA
7. Data analysis
Given the limited data that has been gathered, data analysis is much less rigorous that one might expect from such a complex and ambitious experiment. We do envisage more powerful data analysis being needed for the second stage of the experiment. The small size of the gathered data allows us to include the complete list in this document. The list can be found in Appendix D. Despite the small number of samples, the number of questions allows a wide range of statistical values, correlations and graphical charts to be produced. We present below some statistics that we find most informative and interesting. Questions are translated roughly and abbreviated. A standard 1-5 Likert scale is used for most questions. The exception is Yes/No questions which are treated as 1/0 values.
7.1.
Statistical views of the data

Table 1. Average values of picture clarity.
Question Clarity of picture for the bed Clarity of picture for the building Clarity of picture for the ship
Value 4,58 4,58 1,33
Clarity of picture for the amphorae 1,75
Table 2. Average values of relevance to the point of interest.
Question Relevance for the bed Relevance for the building Relevance for the ship
Value 2,25 2,33 3,83
Relevance for the amphorae 4,08
Table 3. Average values of interest.
Question Interest for the bed Interest for the building Interest for the ship
Value 2,17 2,25 3,4
Interest for the amphorae 2,75
32
EXPERIMEDIA
Table 4. Would pay 1 euro for educational value
Question Educational value for the bed Educational value for the building Educational value for the ship
Value 1,67 1,41 3,25
Educational value for the amphorae 2,75
Table 5. Average values of fun.
Question Fun for the bed Fun for the building Fun for the ship
Value 2,08 2,2 4,42
Fun for the amphorae 3,41
Table 6. Would pay 1 euro for more content like this
Question Content and points of interest similar to the bed Content and points of interest similar to the building Content and points of interest similar to the ship
Value 8% 8% 92%
Content and points of interest similar to the amphorae 92%
7.2.
Interesting observations
Since the focus of EXPERIMEDIA is on users we pay particular attention to Table 5 and Table 6, which probably provide the best measure of QoE. The fun that the participants have is a direct measure of the value they have given to their experience, whilst the question about whether they would be willing to pay for similar services is an indirect measure of the same thing. The first important observation is that people would be willing to pay money for such an application. This is probably the safest way to conclude that the augmented reality component did enhance their experience considerably. Quite interestingly there are huge differences in the average values for the different points. It is clear that points such as the ship and the amphorae have made a bigger impression than for example the bed. Although one can make guesses, it is the goal of EXPERIMEDIA to focus on users. Therefore, instead of attempting to guess which elements of these points make them more
33
EXPERIMEDIA
fun and valuable for the users, we note the observation and hope to have the opportunity to explore the reasons behind it in a focus group during the second stage of the experiment. EXPERIMEDIA aims to correlate QoE measures with QoS measures in order to assess which technical parameters influence the value that the user receives and in which way. Not having integrated the experiment monitoring component in our facility yet, the closest we have to a QoS measure is the participants feedback regarding the clarity of picture. Table 7 allows us to review correlations more easily.
Table 7. Summative data
Point of interest Clarity of picture Relevance Interest
Bed 4,58 2,25 2,17
Building Ship 4,58 2,33 2,25 1,41 2,2 8% 1,33 3,83 3,4 3,25 4,42 92%
Amphorae 1,75 4,08 2,75 2,75 3,41 92%
Educational value 1,67 Fun Would pay 2,08 8%
It is quite interesting, and perhaps shocking, that the QoS and QoE appear to have a negative correlation. Most probably a better explanation would be to conclude that they do not have a strong correlation in this experiment. Even so, it is quite interesting to see that the lower quality of picture that is available at the outdoors location is not enough to make the experience less enjoyable and valuable. Clearly there are other parameters related to these points that have a stronger correlation with the QoE, and it is our goal to explore which are these parameters during the second stage of the experiment.
7.3.
Evaluation
7.3.1. Internal evaluation

First of all we are thrilled to have managed to execute a fully cycle of the experiment, starting from the specification of an abstract scenario and reaching to the point of discussing correlations between QoE and QoS. Clearly, we have implemented a scaled down version and still have a large part of the work ahead of us, but we now have for the first time a practical overview of the process. We are of course to some extent disappointed not to have managed to execute the part of the experiment that runs in the Tholos, particularly when this was the first to be implemented and has already been demonstrated repeatedly to the consortium and to the broader public. But we did manage to gather valuable data and we will use this in the organization and execution of the second stage of the experiment.
34
EXPERIMEDIA
Overall our internal evaluation is that this has been a very successful first stage, and that the accumulated experience will facilitate even further the implementation of the rest of the work.
7.3.2. External evaluation

As promised in D4.3.1, we have sought the independent evaluation of our results from an individual who is not directly linked to EXPERIMEDIA or FHW. Ms Evi Togia fits this description. She holds a masters degree in social sciences and engineering from the University of Manchester. Her independent evaluation is copied in Appendix D as is, without any editing from our side. This independent evaluation is only intended to help us have a more objective examination of our current achievements and outlook and not to affect the project reviewers opinions.
35
EXPERIMEDIA
8. Current status and future plans

Although we have clearly and repeatedly touched upon the issue of the first and second stages of the experiment, we find it fitting to devote at least a few lines to outlining the plans for the remaining of our work on this experiment.
8.1.
Experimental facility
We are currently happy with the technical work done on the augmented reality part of the experimental facility, so we do not expect to be doing and core development work in this direction. We will of course have to examine if and how the augmented reality software that we are using can be linked to the EXPERIMEDIA experiment monitoring component. As far as the data itself is concerned, we do hope to make additions. Specifically, our limited initial testing has indicated that the augmented reality points are too sparse and only some of them are truly interesting and relevant to the items they have been coupled with. We have already asked our content experts (historians and archaeologists) to examine whether more points can be created, given the existing physical exhibitions available in Hellenic Cosmos. We are also quite happy with the technical work done on the video streaming end, and grateful to ATOS for their huge support. We also hope to be able to join this with the experiment monitoring component. We did not manage to integrate the social component with our facility the first time round, but we are now convinced that the updated version that ICCS has circulated will operate as planned. We therefore plan to do work on integrating this component with our venue and with the experiment monitoring component. As has already become available, the integration of the experiment monitoring component with our venue and experiment is a core goal for the upcoming period. As far as the involved hardware is concerned, we hope to be able to have more smart devices for the second stage of the experiment, so that we can run the augmented reality part of the experiment with more participants.
8.2.
Experimental methodology
Although we have only managed to run few experiments with the augmented reality component and none with the video streaming component, we have already had the chance to identify some weaknesses in our methodology and examined scenario, which may call for the corresponding adjustments. Specifically: It is hard to perform extensive tests with the augmented reality component because having only 3 devices we can only have 3 participants at a time. Considering the size of the venue and the dispersion of the augmented reality points, this practically means that only 3 participants may be considered per day, which does not allow for much flexibility. 36
EXPERIMEDIA
Moreover, a single damaged or stolen device can have a major impact on our ability to run experiments. We have already painfully realized that the planned scenario asks for some of our most valuable (and therefore busy) staff members to stop their work for the whole duration of the experiments shows (about 45 minutes each time) in order to be readily available to help. This is clearly not realistic for a large scale implementation. We may need to consider an alternative scenario where an expert is not standing by doing nothing but either has a more meaningful role or is only alerted when there is actual need.
We have not designed a new version of the experimental scenario at this time, nor are we sure that we will, but this is a direction that we will have to examine seriously over the next period if we are to maximize the benefits from the execution of the experiment.
37
EXPERIMEDIA
9. Ethics, privacy, PIA

EXPERIMEDIA will conduct research with human participants and is, in particular, interested in human behaviour and experience with Future Internet technologies to understand how to provide meaningful collective experiences to individuals and society. Given that participants in social and networked media research should have confidence in the experimenters, good research will only be possible if there is mutual respect and confidence between experimenters and participants. As some areas of human experience and behaviour may be beyond the reach of experiments, observation or other form of investigation and may raise ethical considerations, EXPERIMEDIA will provide an ethics management process that incorporates ethical and data protection review of experiments. Appropriate management of ethical issues will be guaranteed by the project management through a mixture of measures to ensure the right technical, physical and administrative environment. The project identified an ethical issues coordinator, as well as a data protection coordinator, which will be incorporated within the overall project management structure. Additionally, an Ethics Advisory Board (EAB) and Data Protection Board (DPB) have been created. The EAB will advise the EXPERIMEDIA consortium on ethical, privacy and data protection issues. The DPB is responsible for ensuring that EXPERIMEDIA is compliant with data protection requirements and that the technical partners develop a system that considers privacy. These are described in more detail in D5.1.1, which also includes the results of extensive work towards the establishment of guidelines that should be followed in the experiments, in order to ensure compliance with ethical requirements and respect for the privacy of those involved in the experiments. In addition to that, D5.1.2 identifies points that specifically the FHW embedded experiment should consider and D2.1.1 details the Privacy Impact Assessment (PIA) methodology that should be followed by all EXPERIMEDIA experiments. In the following we elaborate on how these have been considered and incorporated specifically in the design of this embedded experiment. Given the overlapping nature of the topics and the documents, there may be some repetition of concepts. We keep it in the text by choice, in order to also maintain the point by point reference to the above documents.
9.1.
Minimum ethical principles
In D5.1.1 a set of ethical principles has been identified for the embedded experiments. They have all been considered in the design of this embedded experiment, as explained in the following.
9.1.1. Doing good

The experiment assesses the added value provided to end users by the extension of the Tholos infrastructure and of the Hellenic Cosmos venue in general via exploitation of the
38
EXPERIMEDIA
EXPERIMEDIA technology. Therefore it does good both for those participating in the experiment as well as in general. As far as the participants of the experiment are concerned, they will have the opportunity to benefit from enhanced services that were previously unavailable. In the more general sense, this experiment will be a first step towards making these technologies a part of the normal operation of the Tholos (remote experts participating) and of the venue (augmented reality points scattered in the venue), so that more people can benefit from them in the future.
9.1.2. Doing no harm

Although people working in EXPERIMEDIA will be monitoring closely the execution of the experiment in all stages in order to analyse every relevant piece of information that becomes available, the actual navigation in the virtual worlds is still performed by the properly trained personnel who do that task in the conventional Tholos shows. Their training and expertise guarantees the quality of the experience that will be provided to all participating user groups. For general public the Tholos is meant to provide a feeling of what it was like to live in another era, which will be achieved for the groups of the experiment as well. For a special case of visitors, though, there is a different goal: the students and pupils that visit the Tholos are mainly meant to receive assistance in their history courses. No harm will be done in this direction either, as these user groups are not considered as eligible participants for the experiment. When it comes to the augmented reality component, this merely adds a paper sign to the venue. It makes no difference to those not carrying a specialized device and only offers additional and very relevant information to those who do. Moreover the devices are owned by FHW and no visitor devices are ever tampered with. Therefore no harm is done.
9.1.3. Risk management

As the experiment design was formulated, and now as the two stages of the experiment are executed, risks are constantly analysed, evaluated and treated, in the same sense as in D1.1.2. In the experiments risk register, two types of risks are identified: risks for the participants and risks for the experiment itself. In the context of ethical oversight of the experiment of course, it is the former that is of core interest. The current instance of the risk register for the participants is displayed in Table 8. As can be seen all identified risks have been treated with the AVOID option. In fact this is a strategic choice for the experiment: risks for the participants will be avoided, even if that moves the risk to the experiment itself. In this manner we can be assured that the participants of the experimenters will not be facing any risks.
9.1.4. Consent
The preparatory phase of the experiment involves the explicit communication of any relevant information to the eligible participants (i.e. what the experiment is about, what it entails, which is their role, etc). Only those eligible participants that have agreed and have signed a note of informed consent are considered in the experiment. This consent has a predetermined duration
39
EXPERIMEDIA
of two months. Differently to what was stated in D4.3.1, this consent is not revocable after the submission of the information. The reason is that all information is fully anonymised from its very creation and therefore it is technically impossible to locate and remove the information provided by a specific person.
9.1.5. Confidentiality
During the experiment only the required data is gathered, this data will only be made available to the individuals that are needed to process that data and no part of this data will be disclosed to any third parties. Gathered data is fully anonymised. All data will be purged after the analysis has been completed and at the latest two months after its gathering.
Figure 21. One of the completed questionnaires. No personal data is listed and all options are multiple choice so that no handwriting is required either
9.1.6. Data protection

Following the review of the experiments plan by the EAB we have reached the conclusion that we do not need to follow a rigorous data protection plan.
9.2.
Ethical oversight principles
D5.1.1 has also produced a more detailed set of ethical principles, more customized to the specifics of EXPERIMEDIA and the embedded experiments. These have also been considered and adopted in the design of the experiment, as seen in the following.
40
EXPERIMEDIA
9.2.1. Informed consent

All participants are informed of the research objectives and all aspects of the research that might reasonably be expected to influence willingness to participate. We explain all other aspects of the research about which the participants may enquire. This is done in the preparatory phase of the experiment, before participants are asked to join the experiment, so their decision constitutes informed consent. We do make sure that participants do not feel pressured to take part in the experiment. In fact, in order to make sure that we do not put pressure on people to take part in the experiment, we have decided not to set daily goals for the number of participants, so that the people in charge of recruiting do not become extra pushy when the numbers do not add up. There is not and will not be any payments to participants. We have stated in D4.3.1 that incentive mechanisms used may include the offer of free passes for Hellenic Cosmos exhibitions, but we have not implemented this during the first stage of the experiment. Even if applied, this will not put visitors that accept to participate in any higher risk than that of visitors who do not participate and pay for their passes for the exhibitions.
9.2.2. Deception
We will never intentionally deceive, mislead or withhold information from participants over the purpose and general nature of the investigation.
9.2.3. Data collection

For the first stage of the experiment we did not collect any personal data about the participants. We mentioned in D4.3.1 that we will collect some personal data about participants during the experiment (for example demographics) and this option does remain open for the second stage of the experiment. In that case we will provide participants with any information to complete their understanding of the nature of the research. We will discuss with the participants their experience in order to monitor any unforeseen negative effects or misconceptions, in the scope of the focus groups. We will adopt a principle of data minimisation: only the necessary information will be collected and processed and the information will be stored only for as long as is necessary. It may not possible not to record specific individuals during the second stage experiment, as we may need to keep track of the individuals that have the EXPERIMEDIA devices in their possession, but we will anonymise the data when the devices are returned at the end of each session of the experiment. User profiles will only be stored when the consent of the users is acquired and only for the purpose and lifetime of corresponding experiments session. There will be no commercial exploitation of user profiles.
9.2.4. Withdrawal from the investigation

We will make it plain to participants that they have the right to withdraw from the research at any time, irrespective of whether some incentive has been offered and accepted. The participants
41
EXPERIMEDIA
will NOT have the right to withdraw retrospectively their consent given and to require that their own data be destroyed, simply because with the fully anonymised data it is not possible to do that. Our EAB assures us that this is okay.
9.2.5. Observational research

The experiment will be in part run in Tholos, which is used by user groups. In order to avoid observing individuals that have not given their consent for this, we will try to run the experiments only with pure groups, i.e. with groups that contain only people that have agreed to participate in the project. In any case observing in the context of this experiment does not include any kind of recording and is limited to observing group dynamics during the show. This is something that is done in the Tholos irrespective of the experiment, as the museum educator that is coordinating the show is always observing the groups response and adapts the navigation accordingly. If the technological choice is made to use audio in order to interact with the experts (the current plan is to post to social networks instead), then only the rooms general sounds will be recorded and not each individual separately. In any case, this will only be allowed for the pure EXPERIMEDIA groups.
9.2.6. Data protection regulation

As expected, we do state that all personal data will be captured and processed according to the applicable data protection provisions, such as Directive 95/46/EC on the protection of individuals with regard to the processing of personal data and on the free movement of such data, including Article 29 Working Party 8/2010 opinion, and Directive 2002/58 on Privacy and Electronic Communications) and the Greek data protection legislation that may be applicable. Having said that, we clarify that this is in fact a void statement for the first stage of the experiment due to the fact that no personal data whatsoever was captured. It remains to be determined whether any personal data will be considered for the second stage of the experiment, as the related questionnaires have not yet been prepared.
9.2.7. Consortium partner responsibility

Be sharing the early version of the experiment description with the consortium we invited our partners to participate in the ethical review of our plans. As the experiment design progressed we updated our partners about our plans. Our partners are of course also invited to monitor the experiment either for the sake of acquiring a better of the operation of the technical components they have provided or simply for their information. Any concern that they may have at that time will be considered and treated accordingly.
9.3.
PIA
As was shown in D4.3.1, no further PIA is required. Nevertheless, written consent of a two duration of two months is acquired. Dr. Manolis Wallace will act as the data controller for the 42
EXPERIMEDIA
experiment. Since no other personal data is recorded, the data controllers duty is limited to keeping safe the forms of informed consent (which list the participants names) and deleting them at the end of the predefined period.
43
EXPERIMEDIA
10. Risks
At the beginning of the work on the experiment, two risk registers were formulated: risks for the participants and risks for the experiment itself. These were initially reported in D4.3.1, but were also constantly monitored as a live document, and updated as the project and its environment evolved. In the following we start by presenting how the evolution of the risk registers has affected the work on the experiment to this day, as well as the current instance of the registers with respect the upcoming work.
10.1. Evolution and handling of risks

In the time that has elapsed since the preparation and finalization of D4.3.1, and as work on the experiment progressed, the risks associated with it also evolved accordingly. As was clearly stated in D4.3.1, a firm decision has been made to avoid all risks that concern the experiment participants. This is the main reason that there have not been any changes to that risk register. The experiment's risk register on the other hand has gone through various phases. The most important cause for the evolution of the experiment's risk register was the implementation itself. For example, the D4.3.1 experiment's register contains risk E2, which refers to the risk of required EXPERIMEDIA components not available on time or not compatible with the FHW facilities. As the implementation of the experimental facility progressed it became clear that E2 needed to be partitioned into various distinct items, each one referring to different EXPERIMEDIA components and monitored independently. Thus E2 was reformulated as E5, E6, E7 and E8, each one with low probability and high impact. In the course of the implementation E5 changed to high probability as an unanticipated API update was announced by Twitter and also the first versions of the social component could not be successfully integrated into the facility. As specified in D11.2 high risk and high impact constitute a critical risk, and therefore immediate care was in order for E5. The response chosen was "reduce". The first approach involved providing more solid feedback to ICCS and shifting increased attention and workforce to the integration of the social component. As this approach did not yield the envisaged results in a reasonably short time, and given the fact that the time plan for the experiment was already very tight, this created new critical risk E9 which refers to the delays of implementation and integration leading to inability to complete the work in the time available. The response chosen was "reduce". Specifically, in close coordination with the project consortium it was decided to alter the detailed experiment's detailed time plan by shifting attention to the streaming and AR components for the first stage of the execution and moving the social component to the second stage of the execution. This left the first stage involving only components for which no problems had been observed, thus reducing the probability of E9, and also providing for time for the finalization of the social component, thus also reducing the probability of E5.
44
EXPERIMEDIA
The AR and the streaming component were successfully integrated into the facility and so E6 and E7 changed to low probability. This also reduced the impact of E5 to medium, as enough EXPERIMEDIA components were already integrated into the facility for the experiment to be meaningful. Given that the first stage of the experiment has been completed successfully, the impact of E7 has now been reduced to low, as the corresponding component has already been tested and evaluated to some extent and therefore being forced to exclude it from the second stage of the experiment would be undesired but not disastrous. Among other things, the execution of the first stage of the experiment validated our approach to the recruitment of participants. Thus the probability of E1 was reduced to low for the second stage of the experiment, as following the same approach is expected to yield similar results. We do keep in mind though that larger numbers of participants will need to be achieved for the second stage of the experiment. Overall, the experience of the first phase of the experiment confirmed that the approach to risk management defined in D1.1.2 is applicable and successful and also that the benefits that are gained from its implementation are worth the small overhead that is associated with it. Therefore we have every intention of continuing to monitor and handle risks in the same manner for the second stage of the experiment.
10.2. Current risk registers 10.2.1. Risks for the participants

For the reasons listed above, the current instance of the risk register for the participants is as follows:
Table 8. Risk register for the participants.
ID P1
Description Mobile device damaged during the installation of the mobile application
Probability low
Impact Proximity high Execution phase
Response avoid
Comment Instead of using the participants own devices, the project will supply the mobile devices that will be used during the experiment. See point P1
P2
Mobile device not compatible with the EXPERIMEDIA software Malicious software installed on the mobile device Participants feel pressured to participate in the experiment
high
high
Execution phase
avoid
P3
low
high
Execution phase Preparatory phase
avoid
See point P1
P4
low
high
avoid
This is true particularly for members of groups who visit FHW facilities as part of an agreement between.
45
EXPERIMEDIA
ID Description Probability Impact Proximity
Response Comment We will make it clear to all during the preparatory phase that participation is not a requirement and has no effect.
10.2.2.
Risks for the experiment
For the reasons listed above, the current instance of the risk register for the experiment is as follows:
Table 9. Risk register for the experiment.
ID E1
Description Required experiments cannot be run because 90% of thee visitors are children and adolescents and the agreed Ethical Oversight Measures state we will not be dealing with children Required EXPERIMEDIA components not available on time or not compatible with the FHW facilities Damaged mobile devices
Probability low
Impact high
Proximity Execution phase
Response accept
Comment Already run the first stage of the experiment successfully
E2
low
high
Experiment reduce design
Reformulated as E5, E6, E7, E8
E3
low
medium Execution phase
reduce
We will run the experiment with as many working devices are available. The mobile devices only affect one part of the experiment, and therefore the feedback of participants that are not given a device is still relevant We are planning to execute the experiments at a time that the weather is typically suitable Via close coordination
E4
Weather does not permit the use of outdoors locations
low
low
Execution phase
reduce
E5
Social component
medium
medium Second
reduce
46
EXPERIMEDIA
ID Description not available on time or not compatible with the experimental facility E6 Streaming component not available on time or not compatible with the experimental facility AR component not available on time or not compatible with the experimental facility Experiment monitoring component not available on time or not compatible with the experimental facility Delays in the implementation and integration of the facility make it impossible for the work to be completed on time low Probability Impact Proximity phase of the experiment medium Execution phase accept
Response Comment with ICCS and also by moving to the second stage of the experiment Already integrated successfully into the experimental facility
E7
low
low
Execution phase
accept
Already integrated successfully into the experimental facility
E8
low
high
Execution phase
reduce
Working closely with the technical partners and planning for an early deployment so that there is time to deal with any unforeseen difficulties We have left "dangerous" components for the second stage of the execution
E9
low
high
Execution phase
reduce
47
EXPERIMEDIA
11. Conclusion
This document presents the working report of the 3rd EXERIMEDIA embedded experiment. The document covers all aspects of the experiment, ranging from the purely technical to the purely theoretical ones. More specifically: Relevant elements of the experiment's background have been highlighted in order to improve the readability and completeness of this document, whilst the reader is advised to refer to previous deliverables of the project for further information. Making the descriptions of D4.3.1 more detailed and adopting them to the specifics of the first stage of the experiment, we have presented the architecture of the two parts of the experimental facility and the information flow between the components involved. Continuing, we have presented the various stages of the implementation work that has been carried out. Due to the early timing of our experiment additional work had to be carried out from our part in order to address issues not yet addressed by our technical partners; occasionally not even successfully. Still, in addition to having a working instance of the experimental facility at a very early time in the project, this has also given us the opportunity to provide more concrete feedback to our technical partners regarding their own work and its usability in real life situations. Using the implemented experimental infrastructure and following a reduced version of the scenario already outlined in D4.3.1 we invited a number of participants to experience the EXPERIMEDIA extensions to the venue and provide us with their feedback on them. Unfortunately we were only able to do this for the augmented reality component at this stage, but we are confident we will also cover the other components for the second stage of the experiment. The limited data that has been gathered was analysed and useful conclusions were drawn. Despite the small sample they are based on, these conclusions will be very useful in guiding the preparations for the second stage of the experiment. We reached our conclusions, but we also asked an external individual to weigh in with a second view in order to maintain objectivity. Ethics and privacy have been considered, and constraints have been somewhat relaxed due to the fact that no personal data is kept and all gathered data is anonymised from the beginning. Risks have been monitored and handled throughout the design, implementation and execution of the experiment, following the guidelines specified in D11.2 and using the registers defined in D4.3.1. This process has already proven its worth as various critical risks were identified and handled successfully, protecting the overall experiment of possibly crucial failures. Therefore the monitoring and handling of experiments will be continued into the second stage of the experiment and the know-how gained from it will be shared with consortium partners and particularly those now embarking on the detailed design and implementation of the second year's experiments.
48
EXPERIMEDIA
Overall we are thrilled to have completed this portion of the experiment so soon. We hope this document will be considered during the 1st year review so that together with the experience and feedback we acquired from the worked performed and the questionnaires analysed we can also rely on the project reviewers feedback on how to best proceed for the second stage of the experiment. The experience and know-how gathered while working on this experiment will also permit us to provide more concrete, useful and reliable advice to the new partners who are joining the project now and are planning to execute similar experiments at the Hellenic Cosmos venue.
49
EXPERIMEDIA
Appendix A. Informed consent
50
EXPERIMEDIA
Appendix B. Device lease
51
EXPERIMEDIA
Appendix C.
Structured questionnaire
52
EXPERIMEDIA
53
EXPERIMEDIA
Appendix D. Gathered data
54
EXPERIMEDIA
Appendix E.
Independent experiment evaluation
55

D4.3.2 FHW Experiment Progress Report v1.0

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

D4.3.2 FHW Experiment Progress Report v1.0

Transféré par

Droits d'auteur :

Formats disponibles

D4.3.

2 Experiment progress report including intermediate results

Manolis Wallace (FHW), Pavlos Mavridis (FHW), Anthousis Andreadis (FHW)

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Experimental facility implementation............................................................................................ 16

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Goals and success evaluation criteria

Goal 3: Measure impact of FIRE technologies (Success)

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Experiences offered before EXPERIMEDIA

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Figure 1. Tholos operation without EXPERIMEDIA extensions.

The experience examined in the embedded experiment

3.3.1. First stage of the experiment

3.3.2. Second stage of the experiment

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

4. Experimental facility architecture

Live streaming experiment during the show

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Expert 1 Audio + Video from "Tholos"

Audio + Video from "Tholos"

Audio + Video from Experts

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Augmented reality experiment after the show

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Figure 3. Flow of information and component diagram for the experiment.

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

5. Experimental facility implementation

Streaming experimental facility

Augmented reality experimental facility

Figure 5. The development environment for the mobile application.

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Figure 9. Example of markers used in the augmented reality application.

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Figure 13. Another view of the educators during the experiment.

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Social networks experimental facility

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Monitoring experimental facility

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Before the beginning of the experiment

6.1.1. Deploy the facility

Figure 15. Point 1

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Figure 16. Point 2

Figure 17. Point 3

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

Figure 18. Point 4

Figure 19. Point 5

Copyright FHW and other members of the EXPERIMEDIA consortium 2012

3. Link about Thales

Figure 20. Venue outline

6.1.2. Familiarize ourselves

6.1.3. Inform staff and schedule accordingly

6.1.4. Prepare the informed consent forms