Collaborative Multi-user Augmented Reality Solutions in the Classroom

. Several educational institutions routinely recommend and use Augmented Reality in learning and teaching, for its potential beneﬁts regarding an increase in the students’ motivation and attention as well as their knowledge retention. Most of the time, though, augmented reality applications consist of rather static, single-user experiences which do not fuel collaborative learning. In this paper, we present a novel augmented reality platform whose aim is to facilitate collaboration and interaction between teachers and students by adding synchronisation and multi-user capabilities. We present a case study demonstrating the capabilities of this platform for student-teacher and student-student collaboration.


Introduction
Augmented Reality (AR) is a technology which overlays virtual objects (also called 'augmentations', see [1]) onto the real world, and the virtual objects coexist in the same space as objects in the real world [2]. Despite AR first being introduced in the early nineties as a performance aid for manufacturing airplanes [3], and with precursors dating back several more decades, it is only in recent years that AR applications are more broadly adopted by both industry and academia. This is due to the availability of libraries supporting AR development and the decreasing costs of the required hardware. Especially in education, high-end equipment such as head-mounted displays (HMD) are still to be missed largely, while tablets and smartphones allow to routinely use AR technology in the classroom, accessible by a broader audience.
In the context of Technology Enhanced Learning (TEL), AR provides a multi-modal learning environment, capable of improving users' experiences and increasing the students' motivation [4]. The most relevant advantages of successfully using AR-based applications in the classroom in comparison with traditional learning are [5]: -Decrease of students' cognitive load -Enhancement of spatial cognition abilities -Add the possibility of coherent integration of multimedia -Enabling visualization of intangible concepts -Superposition of information on physical surrounding Despite these advantages, the vast majority of the AR applications used in educational settings offer limited interactivity and collaborative communication capabilities, falling short of providing a multi-user environment that promotes an effective collaboration between students and simplifies the interaction with the teacher. Furthermore, most of the existing applications are stand-alone solutions which use a dedicated architecture. This complicates porting apps to different hardware platforms, extending them with additional content or functionality, or delivering large scale deployment.
In this paper, we present the work done as part of the H2020 ARETE project 3 , towards the development of a software platform for the creation of AR applications with a strong focus on interactivity, collaboration and multi-user capabilities. To facilitate interoperability and simplify adoption of the technology, the platform relies on widely established protocols like Websockets [6], a W3C standard that enables interaction between a web browser (or other client application) and a web server with low overhead, facilitating real-time data transfer from and to the server. We will describe the library that enables multi-user AR and present a demo application showing its capabilities as a tool to easily create collaborative AR applications. All the code related to the work presented in this paper is open source and is freely available on Github 4 .
The remainder of this paper is structured as follows. Section 2 describes the related work. Section 3 outlines the architectural building blocks of the library enabling collaborative AR experiences. Section 4 describes the proof-of-concept application built to show the collaborative and multi-user capabilities of our framework. Finally, Section 5 presents the conclusions and describes future work.

Related work
An early application of AR in education can be found in [7], presenting a tool to visualize mathematical and geometrical objects, using customized HMD and magnetic trackers to track head and pen. It is, however, not until recent years that AR has been seriously considered as an effective educational tool. This has been possible thanks to advancement in hardware: AR applications can now run at decent speed on most smartphones, and there are now several pairs of AR smart glasses on the market. Moreover, software development toolkits such as Unity3D 5 , Vuforia 6 , Google ARCore 7 , and Apple ARKit 8 are now available to 3 arete-project.eu 4 https://github.com/tv-vicomtech/orkestralib-unity 5 unity.com 6 library.vuforia.com 7 developers.google.com/ar 8 developer.apple.com/augmented-reality speed-up application development. There is also growing interest in bringing AR to the web [8] (thanks in part to the development of new web standards like W3C WebXR [9] and libraries like A-Frame 9 , AR.js 10 , or awe.js 11 ).
Several systematic reviews of literature have been published, covering different aspects of using AR in education. They focus on the advantages of using AR [5] [10], the different applications and research trends [11] [12], some specific topics like STEM (science, technology, engineering and mathematics) subjects [13] or serious games [14] [15]. Only a few publications describe collaborative and multi-user AR applications for educational settings, as this functionalities are not implemented by many libraries (Google ARCore only recently introduced support for multi-user AR experiences through Cloud Anchors 12 ). [16] presents the development of an AR book and evaluates its effect on collaboration between primary school students, while [17] and [18] describe applications for AR-based collaboration and compare it with traditional learning. An application of AR for remote collaborative learning is described in [19], while [20] analyzes the effectiveness of AR tools for collaborative learning both in remote and co-located environments.
Outside of education, AR is extensively used, for example, for industry [21] [22] as well as medical applications [23]. Broadcasters have been using AR to embed additional information directly in the shots provided. AR is popular for sports broadcasts [24] but in recent years it is being used in other scenarios [25], where sets specifically designed for AR are used for example for political talk shows or weather forecasts.
-Orkestra: It is the main class, used by each client to specify the server that will provide communication functionalities, the session name (as a server can support several multi-user sessions at the same time) and the user id; -Shared State: the class responsible for defining the state of the variables shared across different users and for the notification of state changes; -Application Context: contains a list of agents, representing the clients connected to the app, and its responsible to send broadcast messages and update the state of variables at an application level; -Agent Context: It is similar to the Application Context, but it doesn't act on a global level and it notifies changes only to a specific client.
When developing an application, the developers must add the code necessary to connect the app with the Orkestra library, specify what kind of variable should be shared across the users and define when the users should be notified about the change of this variable. Even though this library does not provide authoring tools and it still requires to code the desired functionalities, it greatly simplifies the developers work, as it abstracts all the nuances and potential issues of network communication.

Case study: a multi-user geography app
In this section, we describe the proof-of-concept application that we have developed to demonstrate the potential of the library described in Section 3 in the creation of multi-user and collaborative AR applications. We have developed both a Unity client, where the students can compete in a geography quiz, as well as a website where the students and the teachers can connect with any device with a camera, for example a smartphone or a laptop. This demonstrates the interoperability between Unity and Web-technologies in multi-user AR experiences.
The Unity application features two playing modes: in the first one the students plays alone and the task is to correctly map some flags to their respective countries, shown as augmented content on top of the marker (as shown in Fig.  1). Once the player has positioned all the flags to specific points on the map, the app returns a score depending on the accuracy with which the student has positioned the flags. The single player mode is a typical AR application where the user uses the phone to detect a marker, which will in turn display the AR content and lets the user interacts with it.
The multiplayer mode, though, allows the students to play together and jointly complete the assigned task. After typing a username and selecting the multiplayer mode, the student waits for another player to join. When that occurs, each player can point the phone to the marker and start playing. As shown in Figure 2, each player is responsible for positioning the flags of its colour, and the user interface displays the position of each flag in real-time for both players as well as the flag which is currently selected (marked with a larger border). That way both players visualize exactly the same and are aware of the interaction of the other, obtaining a fully synchronized experience. The proof-of-concept we have developed is currently limited to only two users, but it could be easily extended to any number of students, since the Orkestra library has been tested with up to 100 users sharing data without a significant increase of latency.
Apart from the mobile application, we have also developed a webapp showcasing the communication capabilities of our library. In this case, the application allows the teacher to offer a geography quiz to multiple students at the same time. The teacher can send different questions to each user through a web interface and provide immediate feedback. The students will access the AR app by pointing to a specific trigger image with their smartphone, tablet or webcam and will then answer the quiz by clicking on the desired location. The augmented content that each student will see can be selected by the teacher. In our demo the teacher can select a political, physical or topographical map, depending on the type of questions asked. Fig. 3 shows the the interface of the web application, with the interface for the teacher on the left and the one for the student (showing the augmented content) on the center and on the right. The user interface is updated each time a new student accesses the website or new content is sent, either from the teacher or from the students. Questions sent from the teacher appear as augmented content in the student's user interface, while the answer from the students are shown as images in the teacher admin console. Furthermore, depending on the target use  case, the teacher can send questions in a broadcast mode to all the students or individually to a specific one, taking advantage of the features provided by the Application Context and the Agent Context described in Section 3.

Conclusions and Future Work
This paper has presented a library for interactive, collaborative and multi-user AR applications in educational settings. The Orkestra library is actively being developed and its aim is to simplify the creation of collaborative AR experiences for educational settings. As an example of its capabilities, we have implemented a Unity application where two students can participate in a geography quiz, as well as an AR web application where the teacher can send questions to their students (and receive their answers) directly from within the app, thus enabling multi-user remote interactions.
In the next few months, we plan to have a fully working architecture with an API available both for Unity and web-based applications, publish the code and contact stakeholders to gather feedback and promote their usage by content creators. We also plan to include functionalities for video streaming (relying on the WebRTC protocol) to give the possibility to include real time communications functionalities in the applications.
We are also working on creating other demos that can showcase the different innovations brought by the architecture. For example, we are currently developing a Unity application where students can generate songs by combining audio tracks from different musical instruments. Each marker is associated to an instrument and the song generated is a combination of the marker detected on the mobile phone of the different users. Our architecture takes care of synchronizing the information and communicate the types of markers detected on each mobile phone.