Engfi Gate: An Indoor Guidance System using Marker-based Cyber-Physical Augmented-Reality

ABSTRACT


INTRODUCTION
At the early visit, freshmen or new students need navigation guidance to ease them finding their destination room in a building. The common current physical navigation signs are often confusing and even more, make them lost inside the building. This problem makes freshman spend more effort to find their intended destination. In order to address this problem, an interactive guidance system is needed.
In this work, we propose design and implementation of an interactive guidance system that enables interaction between guidance system, building environment, and freshmen. This system provides general information such as maps and images of the building, location-based information like directions, and descriptions of specific places/rooms/facilities for helping new students (or users) explore their department buildings. This system consists of three subsystems: the marker-based cyber-physical interaction (CPI) system, the indoor positioning (IP) system, and the augmented-reality (AR) system. The CPI system uses concept of the cyber-physical system (CPS) which collects sensor data from the physical world and links them to various information sources for real-time analysis in the cyber environment [1]. The IP system produces accurate user position updates so that the system can provide the users with proper and timely location-based information. An AR system has high potential to improve the user experience of applications in which users must access information that has a direct spatial relation to their immediate environment [2]. An Android application, named Engfi Gate, is developed to realize the system design in the test environment of the Engineering Building five at the National Central University, Taiwan. There are several indoor guidance systems [3][4][5][6][7] are proposed in the literature. Some [3][4][5] of them are related to the system proposed in this paper. They are the SmartPDR system [3], the PGPT system [4], and the SumMap+ system [5]. They will be surveyed and compared with the proposed Engfi Gate system to show the proposed system's advantages.
The remainder of this paper is structured as follows: Section 2 introduces three systems that are most related to the proposed one. Section 3 describes the proposed system design. Section 4 elaborates the system implementation, and its comparisons with other related systems. Finally, the paper is concluded in Section 5.

RELATED WORK
SmartPDR [3] is a smartphone-based pedestrian dead reckoning (PDR) system that tracks indoor user's location. SmartPDR employs typical dead reckoning approach, which is widely adopted in localization systems to solve the locating problems for anonymous places in indoor environments. SmartPDR is based on the use of accelerometer, magnetometer, and gyroscope in a smartphone, which are able to observe the pedestrian movement. SmartPDR provides an indoor localization model that computes the displacement through step event detection, heading direction estimation, and step length estimation. The main difficulty of the system is coping with situations that the map database is unknown in advance and available sensors are inaccurate.
The pedestrian guidance for public transportation (PGPT) system [4] is a system for indoor navigation in public transportation transfer buildings. It is an element of a dynamic seamless mobility planning and travel guidance application for public transportation networks of metropolitan areas. Indoor navigation is a key element of multi-modal door to door journeys, especially during transfer events between public transportation vehicles at complex station buildings. Indoor positioning and navigation services are realized by various components of an Android smartphone application. The system integrates the sensed data of cameras, WiFi, GPS, and Bluetooth devices. It also integrates the pressure sensor and the inertial measurement unit for PDR and 2.5D navigation in arbitrary building structures. Maps of multi-level buildings are collected from escape/rescue floor maps and OpenStreetMap data are displayed. It provides indoor routes over multiple levels, elevators, stairs and escalators, which are calculated by the combination of a route search algorithm and a grid based pathfinding algorithm. The overall objective of the system is to accomplish a multi-modal door-to-door journey planning and navigating application.
The SunMap+ system [5] running on Android smartphones can aid users to reach their intended destinations in indoor environments. The system is used as a virtual guide that displays the Sunway University campus 3D floor map and layout via the AR interface. In order to identify the current location of the user in the 3D map, the image recognition technique is applied to initialize and identify locations in the Sunway campus. After user location is determined initially through the visual recognition, SunMap+ continues tracking the user's movement through the PDR scheme, which is regarded as a complementary solution of the image recognition localization technique. Navigation paths to guide the users to reach their intended destinations are also provided along with the 3D floor maps.

PROPOSED SYSTEM
The Engfi Gate indoor guidance system architecture is illustrated in Figure 1.  A marker is something used as a mark or an indication, such as a bookmark and a milestone [8]. The proposed system uses two types of markers: the visible marker and the invisible marker. Visible markers hold data in the format of image-based codes, such as the Quick Response (QR) code, which can be recognized by the camera. Invisible markers hold messages that are sent periodically by wireless communication devices, such as Bluetooth Low Energy (BLE) devices and IEEE 802.11 WiFi devices, and received properly by user devices, such as smartphones. In the CPI system, the role of markers is to bridge the cyber and the physical environments. For example, when a user's device recognizes a QR code or receives a BLE message properly, he/she can be aware of his/her position and related spatial information.
The marker-based CPI system gets inputs from visible markers, such as the QR code, and invisible markers, such as BLE messages sent by the temperature, light, humidity, and sound sensors. So, the system is marker-based and is aware of environmental conditions. Moreover, the CPI system also connects to actuators, so it can control physical devices associated with the actuators. The CPI system also connects to web or game services, which create a virtual environment or web pages corresponding to the physical world. Internet users can roam in the virtual environment or browse the pages to interact with the physical world and the Engfi Gate users. Such interactions are similar to those occurring in mixed reality environments [9]. The web or game services can also access databases, so cyber conditions (e.g., user schedules and profiles) can be retrieved and stored when user interactions proceed.
The IP system utilizes one or more markers, either visible or invisible, to decide the position of the user. On the one hand, when a visible marker is recognized, the user is assumed to have the same position as the marker. On the other hand, when three or more invisible markers are received, the user determines its position by the WCL algorithm. The IP system also uses Dijkstra's algorithm to find the shortest path from the user's current postion to his/her intended destination, as described later in Section IV.
The AR system provides two display modes: the phone mode and the head mounted display (HMD) mode. In the phone mode, the guidance information is superimposed into the environmental images and the user uses buttons on the phone screen to interact with the system. In the HMD mode, user's phone is embedded into a HMD device for users to have 3D view of environmental images. The guidance information looks like floating above the images, and the user use the virtual button to interact with the system.

RESULTS AND ANALYSIS
This section describes a results and analysis of implementation of the above-mentioned Engfi Gate system design, and also shows the comparisons of the system with other related systems.

Hardware/Software Specifications
The hardware/software specifications of the Engfi Gate system implementation are described below. The system is implemented on an Android-based smartphone (Asus ZenFone 2 Laser, ZE550KL, running Android 5.0 Lollipop) supporting Bluetooth 4.0. Some Raspberry Pi B+ devices, equipped with temperature, humidity, light, and sound sensors, are deployed in a building to sense physical world environmental data. The Raspberry Pi B+ devices are also equipped with Bluetooth Low Energy (BLE) USB dongles to send out the sensed data through BLE beacon messages via BLE advertisement packets. Some Seekcy devices (model M0L) [10], supporting BLE 4.0, are also deployed in the building to broadcast BLE beacon messages for the purpose of indoor positioning. VR Box Virtual Reality [11], as shown in Figure 2, is used as the HMD. Unity3D game engine is used to synthesize 3D scenes to be displyed as AR materials. Furthermore, Vuforia SDK [12] is used for AR generation, while Google Virtual Reality (GVR) SDK [13] is used for the HMD generation. The Engfi Gate system also involves a web server using the Codeigniter PHP framework and the MySQL database.

Marker-based CPI System Implementation
With the help of markers, the marker-based CPI system is performed to connect cyber and physical environments. There are two types of markers used in the system: visible markers and invisible markers.

Visible Marker: QR Code
In the system implementation of visible marker, image of a QR code is used. we encode the floor number, X and Y coordinates into a QR code, as illustrated in Figure 3. The encoding is quite simple. There are three values which are separated by separated by the hash sign "#". The first value is the floor number, the second is the X coordinate, and the third value is the Y coordinate. This type of marker is very useful for determining the position of the user.

Invisible Marker: BLE Beacon Message
There are two types of devices to send out the beacon messages as invisble marker. The first device is the Raspberry Pi B+ device equipped with a BLE USB dongle, as shown in Figure 4. Such devices are usually connected with sensors to sense the physical phenomena. The sensed data are embedded into BLE beacon messages to be sent out. The BLE beacon message, called Pi beacon in our implementation, uses the first 11 bytes of the BLE advertisement packet to store sensed data. The arrangement of the 11 bytes is as follows: 4 bytes for the light sensor, 2 bytes for the humidity sensor, 3 bytes for the sound sensor, and 2 bytes for the temperature sensor. On receiving Pi beacons, the user device (i.e., the smartphone) can retrieve physical conditions of the surrounding environment and show them on the display. Figure 5 shows the screenshots showing context information, namely the values of humidity, temperature, lightness, and loudness, on the upper left corner.
Another device to send out BLE beacon messages is the Seekcy device. The position of the Seekcy device is embedded into BLE beacon messages. On receiving beacon messages of Seekcy devices, the user device (i.e., the smartphone) can then know the position of the Seekcy device sending out the beacon message. This type of device is very useful for accurately determining the user position with the WCL algorithm, as will be described later.

Cyber-Physical Communication and Interaction
There are two kinds of users in the Engfi Gate system: physical users and cyber users, as shown in Figure 6. Physical users use smartphones to run Engfi Gate; they walk into the building and their positions are tracked by the system. Cyber users use desktop or laptop computers to access the Engfi Gate web page or virtual environment through Internet. On the web page, cyber users can see physical users in a zone-by-zone manner and can invite one or more physical users to communicate by texting or speaking. On the contrary, a physical user can search for a specific cyber user and invite him/her to communicate by texting or speaking.

Figure 6. The Engfi Gate cyber-physical communication and interaction
The cyber-physical communication and interaction system is useful for physical users and cyber users to communicate and interact. For example, when a new student uses Engfi Gate to explore the department building, old students can invite him/her to communicate for giving advises or instructions. Certainly, the new student can invite an old student or even a teacher to communicate for obtaining advises or instructions. For special events, such as the freshman orientation, the department chair (if he/she cannot show up in person) can invite all new students to communicate for giving them welcome messages by using the privileged broadcast or multicast function through the Internet access.

Indoor Positioning (IP) System
The IP system utilizes either visible markers or invisible markers to determine the position of the user. The positioning procedures are described below.

Positioning with Visible Markers
When a visible marker or a QR code image is successfully scanned by a user device's camera, the system can immediately determine the user's position by retrieving the position information encoded in the QR code. Thus, only one QR code can determine the user's position properly. The accuracy of the positioning depends on the distance that a QR code image can be scanned and recognized successfully by the camera.

Positioning with Invisible Markers
When three or more different invisible markers or BLE beacon messages are received, the weighted centroid localization (WCL) algorithm can be applied to determine the user's position by using the Received Signal Strength Indicator (RSSI) values of the messages. Below we describe the algorithm details.
The WCL algorithm [14] requires the user device to receive at the same time the BLE beacon messages sent by three or more devices. Before describing the algorithm, we introduce some terminologies. Below, we use the term "anchor node" to refer to a device sending out BLE beacon messages; we use "unknown node" to refer to a device whose position is unknown and is to be determined by the WCL algorithm.
After an unknown node receives a BLE beacon message sent by an anchor node i, the WCL algorithm records the RSSI value RSSI i and the X and Y coordinates a i of the anchor node i, where a i = (x i , y i ) is embedded in the beacon message. After receiving BLE beacon messages from n, n  3, anchor nodes, the algorithm computes the weight W i for the anchor node i according to Equation (1). where g is the degree which determines the contribution of the RSSI values. As suggested in [14], g is set as 1. However, we set g as 2 after observing the results of extensive experiments. Afterwards, the algorithm computes the position P of the user according to Equation (2).

∑
(2) Figure 7 shows the screenshot of the Engfi Gate system showing the user position. The four 3D green cubes at the corners indicate the position of anchor nodes, and the man-like object stands for the user position. We perform extensive experiments to measure the positioning accuracy of our implementation of the WCL algorithm in a room with the length of 9 m, the width of 5 m, and the height of 3 m. Four anchor nodes are deployed on four ceiling corners, with coordinates (83,140), (85,738), (389,126), and (428,737), respectively. The value of g is set as 1 or 2 for the sake of comparisons.
We arbitrarily choose 30 positions to perform indoor positioning using the WCL algorithm for evaluating the accuracy of our implementation. Table 1 shows the experiment results. When g is set as 1, the smallest error is 21 cm, the largest error is 188.5 cm, and the average error is 96.55 cm. When g is set as 2, the smallest error is 12 cm, the largest error is 102.5 cm, and the average error is 46.8 cm. We can conclude that setting g as 2 leads to lower errors and thus higher accuracies.

Path Finding and Travel Time Estimation
The Engfi Gate system can find a path for the user to go from his/her current position to the chosen destination. The path finding task is done by applying the Dijkstra algorithm [15]. Each floor of the 3D map of the Engfi Gate system is represented as a grid, which is a graph in which a node has four neighboring nodes. With the help of grids, the path finding job can be achieved. Note that if the user's current position is on the floor different from the floor of the destination, the path finding will be achieved by finding a path on one floor going from the current position to the entry of the elevator or the stairs, as well as a path on the other floor going from the exit of the elevator or the stairs to the destination. After finding the shortest path, the time needed to travel is estimated by calculating the ratio of the travel distance to the speed of the user. Note that the travel time estimation is context-aware in the sense that the system considers the crowdedness of people to do the estimation. The crowdedness can be derived from the number of users in nearby area if most students use the Engfi Gate system. Alternatively, it can be derived by referring to the speed of nearby users. The estimated travel time is helpful for users to decide to hurry up or to slow down when they are walking. Figure 8 is the screenshot showing the navigation information for reaching the destination with the estimated travel time on the upper right corner.

Augmented Reality (AR) System
The AR system is in charge of synthesizing images to be shown on the user device's display. As mentioned earlier, there are two operation modes: the phone mode and the HMD mode. Below we describe how the AR system deal with the two modes.

Phone Mode
The Unity 3D game engine and the Vuforia SDK together are used to synthesize 3D scenes and the graphical user interface (GUI) to be shown on phone display. The screenshot in Figure 5 is for the phone mode. The user interacts with the system by tapping buttons in the GUI.

HMD Mode
In addition to the Unity 3D game engine and the Vuforia SDK, the Google VR SDK is used in this mode to split the phone screen into a left part and a right part. The two parts are for the left eye and the right eye, respectively. Figure 9 shows a screenshot of this mode. This mode needs a head mounted device, such as VR Box Virtual Reality, to operate. The user places his/her smartphone inside the head-mounted device and the user feels like seeing 3D objects with depth information. This is because the user's two eyes see slightly different views which are calculated by the Google VR SDK to generate 3D illusions. Since the user cannot tap the phone which is put into the head mounted device, this mode uses virtual buttons for the user to interact with the system. The user only needs to take his/her finger in front of the camera to push or touch virtual buttons for the purpose of interaction.

Systems Comparison
We compare our Engfi Gate system with existing indoor guidance systems that are most related to our system, namely the SmartPDR system [3], the PGPT system [4], and the SunMap+ system [5]. The comparison results are shown in Table 2. By the table, we can observe that Engfi Gate has many advantages over others. Specifically, SmartPDR and PGPT use 2D GUI, while SunMap+ and Engfi Gate use 3D AR interface and provide 3D maps. Moreover, Engfi Gate is the only system providing the HMD mode and the cyber-physical interaction. SmartPDR does not provide shortest path finding, but the others do provide. Engfi Gate is the only context-aware system, since it considers the context information of crowdedness when estimating the travel time for a user to reach his/her destination.

CONCLUSION
The proposed guidance system Android application prototype named Engfi Gate System has been developed to help freshmen explore Engineering Building V as their new environment. The full demo video of the system running on an Android smartphone can be found in [16]. Compared with related work, this system has richer features and technology systems in the guidance system area. This guidance system consists 3 major components: Marker-Based CPI, IP system, and AR system. A Marker-Based CPI subsystem brings new interaction between the cyber and the physical environment in the Engfi Gate System. This subsystem roles are to connect the cyber and the physical environment. The IP system using QR Codes and Bluetooth beacons achieves a good result. Indoor positioning uses QR codes to do building positioning. When entering a room, Bluetooth beacons with WCL algorithm performed to gather a good accuracy and real-time positioning. The room positioning needs more than two Bluetooth beacons achieves 46.8 cm average accuracy error. Lastly, the freshman physical position information is sent to Engfi Gate system to update their positions. AR subsystem is implemented because Engfi Gate system runs as the guidance system in the spatial environment. There are two operational modes in this system: Phone and HMD mode. Phone mode helps a new student explore building withholding their smartphone. HMD mode makes simpler interaction by placing smartphone inside HMD device without holding the smartphone. Both operational modes have its own advantages and disadvantages for freshmen.