Virtual Inertial Sensors with Fine Time Measurements

Inertial sensors embedded in mobile devices, such as accelerometers and gyroscopes, have shown great potential to study human motion. In this paper, we propose to estimate the device movement without any access to physical inertial sensors in the mobile. Our idea is to infer the movements of the mobile through radio measurements, a concept we call “virtual inertial sensors”. We propose a method for estimating the rotation of a user that uses only WiFi Fine Time Measurements (FTM) to infer the rotation speed. We evaluate and demonstrate the proposed approach with experiments, using commodity 802.11ac Access Point (AP)s for Channel State Information (CSI) and FTMs measurements, and a Google Pixel 3 smartphone as mobile terminal. While FTM works with only one single antenna, it achieves better performance than a CSI-based estimator that exploits four antennas and multiple sub-carriers at the AP, but is limited by the typical one single WiFi antenna at the smartphone side. Together with walking speed estimation of a user, we envision that virtual inertial sensors can be leveraged by location systems and sensing mechanisms, including 5G, to improve localization accuracy, infer user behavior, and design better and more secure communication.


I. INTRODUCTION
Inertial sensors in smartphones have been exploited extensively in the past years for applications such as dead-reckoning aided localization, fine-grained gesture and posture recognition, and mobile gaming [1], [2], [3], [4], [5]. Smartphone sensors such as gyroscopes, compass, and accelerometer have received particular attention, and they can nowadays operate efficiently in terms of energy consumption. However, they can be affected by significant noise and error compared to industry grade sensors. More importantly, the wireless infrastructure does not have direct access to inertial sensors in mobile devices, calling for the installation of dedicated applications in the mobile device responsible to serve as a collector of sensor measurements. Such a mobile app is usually also responsible to process inertial sensor data, as continuously transmitting raw input data to the infrastructure would result in an unacceptably high network traffic load.
In this paper we aim to answer to the following question: can the infrastructure, and in particular the WiFi APs infer information such as rotation speed (similar to what a gyroscope would measure) and acceleration (similar to what an accelerometer would measure) of the mobile device without access to its physical inertial sensors, using only a single antenna? We call these mechanisms virtual inertial sensors of mobile devices, as the infrastructure performs radio measurements to estimate the information a mobile would typically extract from the inertial sensors.
A straightforward idea is to estimate the mobile's acceleration using ranging measurements. We confirm in the paper that this is indeed possible, showing how well the acceleration can be estimated with commodity devices. What is more challenging is the question of how to achieve a reliable rotation estimate. As the rotation speed is related to angular information, [6], [7] exploit CSI and map it to the angle. In these past works, phase information is used to estimate parameters such as moving distance, heading direction and rotation angle. CSI has received significant attention in the research community in the past years to address problems such as indoor localization and estimating context information of users. However, with only AP-side CSI it is not possible to reliably estimate the rotation speed of mobile devices with a single antenna as typically found in smartphones. As antenna size is fundamentally limited by Maxwell's equations and not by Moore's law, we expect that future generations of mobile devices will experience similar issues. We show in this work that, because of this limitation, CSI measures angle variation using the AP as reference system instead of the client, resulting in an error that increases with the range between the AP and the mobile device.
We then present a solution to estimate mobile device rotation that relies on FTM, a protocol that has been standardized in the IEEE 802.11-2016 standard, with corresponding devices now entering the market. FTM allows to estimate the distance between the AP and the mobile, even without associating the mobile to the AP. We retrieve FTM measurements from a commodity AP working in an 80 MHz channel communicating with a Google Pixel 3 smartphone, and show that FTM can serve purposes beyond the original scope of estimating the distance. This is feasible though FTM does not take advantage of the multiple antennas of the AP and the multiple subcarriers of the spectrum. Furthermore, in comparison to physical inertial sensors, it is not affected by dead reckoning, since it relies on absolute distance estimates to avoid error accumulation over time. Furthermore, our proposed solution does not require the position of the APs deployed.
The rest of the paper is organized as follows. We start introducing potential applications of our idea in Section II.
We then study the simple case of estimating the acceleration using FTM samples (Section III) and then present a geometric model that maps FTM samples to angular rotation, applying signal processing theory in order to provide an estimator of the rotation speed (Section IV). We perform an experimental comparison of performance of an FTM-based estimator versus a CSI-based estimator using commodity 802.11ac chipsets, demonstrating the potential of the proposed approach (Section V). We finally draw conclusion and discuss the main findings of the experiments conducted in this work (Section VI).

II. APPLICATIONS
Using radio measurements as virtual inertial sensors has several interesting applications: Secure localization. Several methods for positioning rely on the inputs from inertial sensors provided by the mobile device. However, these inputs could be corrupted or hacked. An independent method to verify the correctness of these inputs can be derived using virtual inertial sensors, using radio measurements. Virtual inertial sensors can also be used in absence of inputs from inertial sensors, for instance in case these data cannot be trusted for positioning or are not accessible.
User behavior. The WiFi infrastructure can infer the movements of the mobile device as proposed in this work and exploit this information for purposes such as crowd profiling and gaming without requesting the installation of any app to get access to the inertial sensors of the mobile device.
Robust communication at mm-wave. After beam training, continuous beam adaptation is necessary due to environment variations or mobility and rotation of the mobile device (UE, user equipment). This is done to avoid link quality degradation and to maintain a high data rate. For instance, [8] reduces the beam search latency under mobility using data from gyroscope sensors in order to estimate any variation of the angle and then adjusts the antenna sector accordingly. Relative rotation information as provided in this work can facilitate beam maintenance, and provide insights on why the link was disrupted.
Use of virtual inertial sensors in 5G. In addition to the use of virtual inertial sensors for WiFi systems, our system can also be applied to 5G cellular systems. Cellular localization is a Radio Access Technology (RAT)-dependent positioning method, while other technologies that are studied in 3GPP standards are called Radio Access Technology (RAT)independent positioning, and WiFi is one of them. Although there is standardization work in 3rd Generation Partnership Project (3GPP) to use WiFi measurements, one should consider that the location server in 5G may or may not have the information about the Wi-Fi AP exact locations as they are not part of 5G infrastructure. The latter case may limit the usefulness of WiFi measurements for 5G applications.
In this work, we investigate what other information can be extracted using WiFi measurements that do not rely on the knowledge of the AP position, and introduce the concept of virtual inertial sensor. According to this view, the 5G location  In the figure, we show an example of communication from a gNB that sends a request to the mobile device to collect FTM measurements. The collected data can be used for virtual sensor inference. server (called Location Management Function in 5G) can request round-trip-time (RTT) measurements to the mobile device through the radio network node (gNB), which, in turn, will perform FTM measurements to one of the APs in range. After the measurement period, the mobile sends the report to the location server, which will be responsible to extract virtual inertial sensor data. An example of such an exchange of information is shown in Figure 1 where a 3GPP compliant protocol (New Radio positioning protocol, NRPP [9]) is used for the communication between the location server, gNB and the mobile.
In relation to the applications mentioned above, trusted access to inertial sensors is also relevant for cellular networks, as 3GPP has defined procedures, protocols and messages to support accelerometers and gyroscopes in Long-Term Evolution (LTE) [10], and these procedures, protocols and messages are envisioned to be extended to 5G networks [11]. In the same way, inference of user behavior for context-aware applications and support of mm-wave networks are also potential topics in the context of 5G networks.

III. BACKGROUND AND ACCELERATION
We first introduce the FTM protocol of the IEEE 802.11mc amendment, included in the IEEE 802.11-2016 standard, and then evaluate the simple case of acceleration estimation as first example of a virtual inertial sensor.

1) Background on Fine Time Measurements:
The FTM protocol, standardized by IEEE 802.11-2016, allows to estimate the distance between a pair of WiFi devices, and it is now available in commodity chipsets. In order to initiate the FTM process, an FTM initiator (FTM-I) sends an initial FTM request to the FTM responder (FTM-R). The FTM-I is typically a mobile device, indicated in this work as User  Equipment (UE). The FTM-R is the corresponding AP. Both devices, AP and UE, must support the FTM procedure. If the FTM-R agrees to perform measurements, it sends a FTM message at t 1 that arrives at t 2 at the FTM-I. The latter then sends the ACK message to the FTM-R at t 3 . After that, the FTM-R receives the ACK at t 4 . The distance between FTM-I and FTM-R can be then calculated as: where c is the speed of the light and RTT is the Round Trip The FTM protocol allows to increase the number of samples per burst, that is the number of FTM/ACK exchange used. The final reported RTT is the average over the number of messages in the burst. In order to understand the accuracy of FTM for distance estimation, we evaluate the distance accuracy at different distances and with different burst sizes. The results are plotted in Fig. 2. In case of only one sample per burst, the results show that for all cases we achieve a median ranging error of 0.5 m. The figure also highlights the benefit of a higher number of samples per burst, confirming the results presented in [12]. However, the RTT for each pair of FTM message/ACK is not available in the report, but only the average RTT over the burst with the corresponding distance.
2) Acceleration: After presenting how FTM measurements are generally used and how they perform, we then briefly evaluate the simple case of estimating the user walking speed using only FTM measurements. The study focuses on understanding how reliable this estimation is and understanding the basic formulation for computing a virtual sensor. The mobility component of the UE can be identified at a low frequency using the Fourier Fast Transform (FFT) of the distance samples sequence: where F indicates the Fourier transform and F (f ) the frequency transform of the distance d(t).
In the transform domain, we calculate the frequency at which F (f ) has the main peak, f P EAK (with unit [1/s]). We then estimate the walking speed as: where d norm is the difference between the maximum and the minimum of estimated distances in a time window of 1 s. We conduct a test to evaluate this methodology. In this test, a human carries the mobile phone moving toward from the AP at an average speed of 0.5 m/s. Fig. 3 shows the Empirical Cumulative Distribution Function (ECDF) of the speed error in m/s, reporting a median error of around 0.047 m/s.

IV. MOBILE DEVICE ROTATION
For the estimation of the mobile device rotation, we first define the scenario under study, then explain why CSI cannot provide reliable rotation speed and then present our solution based on FTM measurements.

A. Scenario
For a human carrying the mobile, let us assume that the elbow is close to the torso, with the right upper arm (which extends from the shoulder to the elbow) parallel to the torso. The right shoulder of the human serves as the origin of the Z axis and the radius δ z is the length of the forearm, which extends from the elbow to the hand carrying the mobile and it represents the distance between the mobile and its rotation axis. An illustration is presented in Fig. 4. Nevertheless, the system works also without the assumption that the elbow is close to the torso, as long as the changes in the radius δ z can be considered negligible during the measurement.

B. Exploring CSI for rotation speed estimation
A possible solution is to leverage CSI measurements at the AP, [13], [14], [15], [16], [17], [18], [19]. With a WiFi chipset like the one used in this work (see the details in Section V), we have four antennas and 234 subcarriers. From CSI, we can z Fig. 4: Illustration of scenario with human carrying the mobile, and rotation movement of the UE along a circular trajectory (rotation axis: shoulder/upper arm). infer the Angle Of Arrival (AOA) of WiFi signals from the UE to the AP. A fundamental aspect of the rotation is about the rotation axis. In fact, detecting the UE rotation using AOA information causes failures in the mobile rotation estimation as CSI estimates the rotation using the AP as reference system instead of the client. In other terms, it estimates the angle rotation of the client as seen from the AP. More in details, let us call the mobile as User Equipment (UE) and the origin of the reference system as UE 0 . Applying CSI works well only when the UE rotates around the AP, with the AP at the origin of the Z axis UE o . However, this does not happen in practice. The worst case occurs for mobile device faraway from the AP: rotations with radius δ z can be hardly sensed from the AP relying on the AOA, and the estimator would declare the device as static or quasi-static. Fig. 5 illustrates the estimated angle AOA and how sensitivity changes with the ratio of the distance d to the radius δ z .
Based on the above explanation, in the next subsection we introduce our method to infer the rotation speed of the UE performing measurements from the AP.

C. From FTM ranging to rotation
We present a method for estimating the rotation of the UE using only distance measurements performed from the AP. Our insight is that a few snapshots of FTM measurements can give us also information about the relative rotation of the UE besides distance information. We will show that this can be done even if FTM does not take advantage of the multiple antennas and subcarriers of the hardware (in contrast to CSI).
A schematic representation of the problem is presented in Fig. 6. We define the relationship between the FTM ranging measurement d and the rotation speed ω as: where d 0 = p AP − p UE0 . In the ideal case where the ranging measurements are not affected by measurement noise and multipath, Eq. 4 can be represented with a harmonic function of period T = 1/ω and amplitude 2d 0 δ z . However, in a real case of rotation, Eq. 4 is the sum of different harmonics at different frequencies, and the rotation component usually becomes evident in the frequency domain. For this reason, we propose the following method. We gather FTM ranging measurements d(t) over a time window of length L and we then compute the FFT of the sequence: where F indicates the Fourier transform. We then remove the DC component (caused by d 2 0 ), and process the resulting sequence in the frequency domain. We investigated several options to process the sequence, and we present here the best performing one.

D. Criterion to infer the rotation with noisy data
In order to choose the best option to process the sequences, we place the UE at 5 m of distance from the AP, in an open area. The UE is mounted on top of a Kobuki Turtlebot II robot running ROS (Robotic Operative System), producing two rotation speeds of 7 and 70 • /sec. More details about the experimental platform will be provided in Section V-A. For each experiment, we first gather FTM samples in a time window of L=60 seconds and we then compute the F(f).  shows the single-sided amplitude of the normalized FFT for both cases, 7 and 70 • /sec, where we can clearly see the rotation component of the UE (true peak in the figure) moving at low frequencies. Calculating the f P EAK , as introduced in Section III-2, we can estimate the desired rotation speeds of the UE. The problem of the proposed approach f P EAK is that the waiting time for obtaining an estimation becomes quite large (L=60 seconds). On the other hand, as shown in Fig. 7b, reducing L up to one second makes the sequence very noisy and the f P EAK approach unusable. For this reason, we introduce a criterion to estimate the rotation frequency in a time window of L=1 second, in presence of noisy data. For such short sequences, we compute the FFT of the upsampled sequence d (t), increasing the rate by a factor N (for a review of upsampling process see e.g. [20]). Upsampling allows us to smooth out discontinuities in the sequence, specially for small L values. The idea of the criterion then is to integrate the area of the F (f ) after DC removal. Our intuition is that, since in such a very short time window the detection of a small variation of the rotation speed becomes complicated, compared to higher values, the noise specially affects small values of rotation and so the range of small frequencies. For this reason, the higher the rotation speed, the faster we expect that the area increases as a function of the frequency f . This intuition will be confirmed in the experimental evaluation in Section V and Fig. 9.
More formally, we integrate F (f ) after DC removal and calculate f 1/2 , the frequency at which the area of the normalized FFT is equal to 1/2: where f 0 is the first frequency component after the DC component and F tot is the total area of the FFT after DC removal. We compute the absolute value of the Pearson correlation coefficient of the experimental set of f 1/2 with respect to the real rotation speeds. The result is a correlation of around 0.6 (0 means there is no relationship between the variables, while 1 means that data lie on a perfect straight line), for different distances. We compute the linear regression of the experimental f 1/2 with the respect to the real rotation speed, and we can then estimate the rotation speed, asω where m and q are the slope and the bias of the linear regression, respectively. Eq. 6 shows that we are able to estimate the rotation speed without knowing the radius δ z , thanks to the fact that we analyze Eq. 4 in the frequency domain. We note that the methodology works as long as δ z > 0, which holds in the scenario defined in Section IV-A.
We also note that, as the proposed method does not only use the dominant peak in the frequency domain but the whole frequency spectrum, the proposed method will provide an averageω in case of noise in the estimate. We finally evaluate the accuracy on the estimation of the rotation speed in Section V.

V. EVALUATION
In this section, we compare the proposed rotation estimator with another solution that exploits CSI. For this purpose, we first introduce the experimental platforms and then we analyze the performance of both estimators, the proposed one obtained from FTM and the one from CSI. In both cases, we use a Google Pixel 3 as mobile device.

A. Experimental Platform
As FTM-R, we use the fitlet2-CJ3455 1 platform with the WiFi Intel 8265 chipset and the WiFi Indoor Localization Device (WILD) configuration tool which enables all the configuration files. As FTM-I for time measurements, we choose the Google Pixel 3 phone with Android Pie (API Level 28) and have the location-based services enabled at the system level. Furthermore, in order to access the FTM protocol and to initiate the measurements, we install the android-WifiRttScan application in the phone. More specifically, we modify its code for data collection and for the configuration of the number of measurements per packet. Once the application starts, it lists all FTM-Rs and by selecting one of them, RangingRequest is initiated via WifiRttManager. The main activity of this code then displays and stores information from the selected FTM-R, including its distance from the FTM-I.
Regarding the collection of Multiple-Input Multiple-Output (MIMO) CSI data, we use instead a QHS8405S4-RDK device, the Quantenna (QTNA) 4x1 Uniform Linear Array (ULA). 1 https://fit-iot.com QTNA device supports 802.11a/n/ac protocol, 20/40/80 MHz bandwidth, and frequency range from 5.15 GHz to 5.85 GHz. It supports extracting up to 4x1 channels and all CSI data from the driver are accessible over Transmission Control Protocol (TCP) sockets. As UE, any WiFi device can be used.

B. Rotation
We perform experiments in a corridor of an office area, where multipath can occur. As shown in Fig. 8, we mount the UE on top of the Kobuki Turtlebot II robot running ROS (Robotic Operative System), in order to run experiments at different rotation speeds. More specifically, the rotation mechanism of the robot is connected to a RaspberryPi where we run a Python script in order to control its rotation speed. AP and UE are positioned at distance of 1, 5, 10 and 15 m from each other. In case of rotation, the robot produces different rotation speeds, namely 7, 33 and 70 • /sec. For each experiment, we gather measurements over a second window.
FTM. After collecting a set of around 100 FTM ranging samples in a window of 1 second, we apply the methodology presented in Sec. IV-D, setting m=0.02 and q=-0.0445 for the linear regression of Eq. 6. We upsample by a factor of N = 10. Based on our experience, N = 10 provides a good tradeoff between performance and higher processing time. Fig. 9 shows the normalized F (f ) as a function of the rotation speed ω, from a single snapshot at 5 m. The FFT analysis shows that the frequency f 1/2 decreases with the speed of rotation, as confirmation of the intuition explained in Sec. IV-D. We then evaluate the accuracy of the proposed method for rotation speed estimation, increasing the FTM burst size at 5m only. Fig. 10 shows that the Root Mean Square Error (RMSE) increases using 1, 3, 10 and 50 samples per burst. The reason is that, averaging several samples for moving targets results in information loss and so, as we consider moving devices (accelerating or rotating), we focus only on RTT measurements with burst size equal to one sample.
We then evaluate the accuracy of the proposed method for rotation speed estimation. Fig. 11 shows the ECDF of the rotation speed error of all cases, at 1 m (11a), 5 m (11b), 10 m (11c) and 15 m (11d). In all cases, for high ω (70 • /s) the method loses in accuracy. Yet, as shown in the figure, we are able to estimate the rotation rate with reasonable error in one second, especially for low speeds. In particular, we achieve a median rotation error of less than 3 degrees for low rotation speeds for all distances under tests.
CSI. Also for this evaluation, we collect around 100 AOA samples in a window of 1 second. We leverage the well known MUltiple SIgnal Classification (MUSIC) algorithm to estimate the strongest AOA, which is a technique widely used in the literature [21], [22], [13]. More specifically, we calculate the MUSIC spectrum from the autocorrelation matrix and we consider the highest peak as estimator of the strongest AOA. We then apply the same methodology presented in Sec. IV-C, as for the FTM samples. The Fig. 11 shows the ECDFs of the obtained rotation speed error for all four cases, at 1 m (11a),   We analyze the results in more details in the bar plot in Fig. 12, where we show a comparison between the median estimates of the FTM and CSI approaches. The "ground truth" is further plotted, which indicates the true rotation speed as set on the Kobuki Turtlebot II robot. Fig. 12 shows that the performance of CSI degrades more significantly as the rotation speed increases. CSI shows dependence on the distance between the AP and the UE, confirming our finding that performance become poor at larger distances (cf. Section IV-B). Instead our proposed method based on FTM provides robust performance as the speed and distance increase. Given that FTM can also be used to estimate the acceleration of the mobile device (cf. Section III-2), our FTM-based solution can be used standalone as virtual inertial sensor for estimating both acceleration and rotation speed.

VI. CONCLUSION AND DISCUSSION
This paper presented the concept of virtual inertial sensors, where the infrastructure estimates parameters that characterize the movements of a mobile device without any access to physical inertial sensors, just using radio measurements. This concept is applied to WiFi systems, but, thanks to the support of collecting WiFi measurements in 5G standards, it is also applicable in 5G networks. We have presented a model that uses only WiFi FTM to infer the rotation and the speed of a user. We have shown that CSI is not suitable to estimate the rotation of mobile terminals as it operates on a wrong reference system to estimate the rotation (as confirmed by our experiments). It also cannot be used to estimate the mobile device acceleration. We have evaluated the proposed approaches with experiments, demonstrating a median rotation error of less than 3 degrees for low rotation speeds, and a median walking speed error of around 0.047 m/s. As discussed in the paper, the main limitation is that the proposed method requires that the device is held and rotates around an axis such as human upper arm. Rotations around the center axis of the mobile device cannot be detected yet. There are several aspects that must be addressed to improve the practicality of the proposed methodology. First of all, experiments have been conducted using a robot, and the question that must be addressed is the validation of the results using humans. A second open question is to perform studies of rotation under mobility to study how well we can separate the rotation component from the acceleration. A third aspect it to understand how to estimate more complex movements of the mobile terminal. Finally, we envision that by fusing FTM and CSI we can leverage some of the benefits of both metrics to design more robust virtual inertial sensors.