Open-source software tools for measuring resources consumption and DASH metrics

When designing and deploying multimedia systems, it is essential to accurately know about the necessary requirements and the Quality of Service (QoS) offered to the customers. This paper presents two open-source software tools that contribute to these key needs. The first tool is able to measure and register resources consumption metrics for any Windows program (i.e. process id), like the CPU, GPU and RAM usage. Unlike the Task Manager, which requires manual visual inspection for just a subset of these metrics, the developed tool runs on top of the Powershell to periodically measure these metrics, calculate statistics, and register them in log files. The second tool is able to measure QoS metrics from DASH streaming sessions by running on top of TShark, if a non-secure HTTP connection is used. For each DASH chunk, the tool registers: the round-trip time from request to download, the number of TCP segments and bytes, the effective bandwidth, the selected DASH representation, and the associated parameters in the MPD (e.g., resolution, bitrate). It also registers the MPD and the total amount of downloaded frames and bytes. The advantage of this second tool is that these metrics can be registered regardless of the player used, even from a device connected to the same network than the DASH player.


Introduction
In multimedia systems, it is essential to know about the required resources for and effective deployment and usage, as well as about the Quality of Service (QoS) that is, or can be, offered, to the customers.
On the one hand, many multimedia applications are resources intensive, so it becomes very valuable to accurately know about the resources that are actually used to take the corrective actions and/or to choose the adequate processing devices for seamlessly running these applications. A typical example is the execution of media players for traditional and/or immersive formats. In particular, in Virtual Reality (VR) scenarios, in which many heterogeneous media elements and advanced interaction features can be integrated (e.g. [1] [2]), the hardware and software requirements of the processing devices become key aspects to take into account. Therefore, appropriate tools are needed to accurately measure resources consumption metrics (e.g., CPU, GPU, RAM, data exchange…). These tools can be used, for instance, to benchmark specific applications, to determine the impact of the inclusion of different media elements and features, and to select and/or adapt the processing devices with the required hardware and software resources to guarantee satisfactory QoS and Quality of Experience (QoE) levels.
On the other hand, multimedia streaming services, like those making use of Dynamic Adaptive Streaming over HTTP (DASH) [3] [4], raise key challenges to provide satisfactory QoS and QoE levels to the consumers. That is especially true when distributing high-quality media over non-controlled network environments, and when using equipment with limited resources. DASH provides key advantages to contribute to this, as the client devices can dynamically select the available media quality that best fits the current network and consumption device status. In the last years, the research community and industry are devoting significant efforts to overcome remaining challenges in DASH Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned streaming scenarios. Relevant examples are: bandwidth prediction (e.g. [5]); latency minimization (e.g. [6]); design of dynamic media quality switching strategies (e.g. [7]), scalability (e.g. [8]); and QoE modeling (e.g. [9]). All these research objectives and initiatives share a common denominator: it is essential to measure and register relevant QoS and QoE related metrics to better understand what are the limitations, propose appropriate solutions and corroborate the obtained performance and benefits, probably in comparison with benchmarking or alternative solutions.
The availability of media players with appropriate probes and Application Program Interfaces (API) for measuring valuable metrics is of great value to succeed in all these goals. The most relevant example is the DASH Reference Player (dash.js) 1 , which comes with a quite complete and handy API for measuring valuable metrics, and even with tools for visualizing them. However, unfortunately these tools cannot be also always used and/or may not completely meet the targeted requirements. In some situations, other players are being used and cannot be replaced. These other players may not provide tools for measurement of metrics, and may not be open-source. Therefore, although they may be able to play out the desired DASH streams, they may be considered as black boxes that prevent from actually measuring key QoS and QoE related metrics. In some other situations, like when using Unity based scenarios, the dash.js player cannot be integrated, and not many compatible DASH players exist. Likewise, it can be beneficial to obtain further relevant metrics and information about the DASH streaming session beyond the ones provided by dash.js. Examples are: the round-trip times since request to full download for each DASH chunk; the number of TCP segments and bytes for each chunk, and information available in the Media Presentation Description (MPD) metadata file describing the DASH assets (e.g. resolutions, rates….). In addition, the availability of an external tool to measure and register the targeted metrics can be very beneficial to compare the performance and behavior of different players for specific DASH streams.
This paper presents two open-source software tools that have been developed to provide effective solutions to the discussed needs and limitations in terms of metrics measurement. First, a tool for the measurement of Resources Consumption Metrics (called RCM-UV tool, where UV stands for the University of Valencia, entity having developed it) has been developed to evaluate the performance and requirements for different Windows applications. The RCM-UV tool is able to measure the usage of CPU, GPU, RAM for any Windows program (i.e. process id) by just providing its name. Unlike the Task Manager, which requires manual visual inspection for just a subset of these metrics, the developed tool runs on top of the Windows Powershell 2 to periodically measure these metrics, calculate statistics, and register them in log files. The rationale behind developing the RCM-UV tool was to analyze how many resources are 1 Open-Source DASH Reference Player (dash.js): https://reference.dashif.org/dash.js/ Last Access in February 2020 used/necessary to simultaneously play out different combinations of media elements in ad-hoc Unity-based and web-based players. However, their applicability spans to other media scenarios and applications.
Second, a tool for the measurement of QoS and QoE related metrics from DASH streaming sessions (called DQM-UV tool, where again UV stands for the University of Valencia, entity having developed it) has been developed to evaluate the performance and behavior of media applications making use of DASH streaming over non-secure HTTP connections. For each DASH chunk, the tool registers: the round-trip time from request to download; the number of TCP segments and bytes; the effective bandwidth; the selected DASH representation; and the associated parameters in the MPD (e.g. resolution, frames per second, bitrate…). The MPD and the total amount of downloaded frames and bytes are also registered. The rationale behind developing the RQM-UV tool was to analyze the obtained QoS, and infer the perceived QoE, in different DASH streaming scenarios, considering both traditional and immersive media formats, via different network connections, under different conditions, and when using different media players and Adaptive Bitrate (ABR) strategies. The advantage of this tool is that these metrics can be registered regardless of the player being used, even from a device connected to the same network than the player.
Both software tools are completely operational and have been released on Github (https://github.com/ETSE-UV/), such that the community can take advantage of their existence, and customize them to the particular needs, if necessary. A detailed documentation about how to run and use these tools is also provided in the Github repository.
The remainder of this paper is organized as follows. Section 2 provides a brief review of related tools and works. Then, Section 3 provides more details about the developed tools. Finally, Section 4 presents some conclusions and plans for future work.

Related Work
The measurement of performance and QoS metrics has been traditionally a fundamental aspect when designing and deploying multimedia systems. This section briefly indicates existing practices, and reviews tools and research works, in this domain.

Resources Consumption Metrics Measurement
In Windows applications, the usage of resources consumption metrics is typically determined by relying on the Task Manager, and by taking snapshots or manual annotations at certain times. Although this process could be somehow automated by developing routines and making use of Optical Character Recognition (OCR) tools, it is still not very flexible and convenient. In addition, the Task Manager provides valuable information about some metrics, but may not inform about all the metrics of interest, and/or about their associated parameters or information (e.g., used GPU engines).
If the specific application at hand is open-source or developed from scratch, then specific Operating System (OS) tools and libraries could be used to measure the desired metrics. However, this situation does not always happen, but the measurement of these metrics when using (closed) third-party applications is often desired. Therefore, the development of an external tool for these measurements allows overcoming the above limitations, while decoupling the execution of tasks from the measurement of metrics.

DASH Metrics Measurement
In DASH streaming scenarios, the measurement and analysis of QoS and QoE related metrics is commonplace. For instance, this is typically used to determine the benefits of novel techniques to improve the performance in terms of the challenges introduced in Section 1. Most of the existing works in these domains (e.g. [5][6][7][8][9]) make use of the APIs provided by existing DASH players, like dash.js or the Bitmovin Analytics API 3 , or simulation tools, like Network Simulator 3 (ns-3) 4 . Other recent works have built endto-end evaluation testbeds, by relying on these players and APIs and extending them with registration and analysis features (e.g. [9], [10]). However, the same limitations discussed for the measurement of resources consumption metrics apply to the measurement of DASH metrics. On the one hand, it may happen that the available measurement tools do not provide all metrics of interest, like the size and round-trip times for each DASH chunk. On the one hand, the available players and APIs cannot be always used, but closed third-party players need to be used or analyzed. Therefore, the availability of an external and flexible tool for such purposes becomes very useful.

Performance / QoS Metrics Measurement Tools
This section describes the RCM-UV and DQM-UV tools, including their objectives, requirements, key implementation aspects, and the necessary steps to make use of them. Both tools are console based, and have been developed by using Microsoft Studio, in C# language.

Objectives.
The RCM-UV tool has been developed to be able of measuring and registering relevant resources consumption metrics when running either single process or multi-process Windows programs, given their process name. In particular, it registers the percentage 3 Bitmovin Analytics API Reference: https://bitmovin.com/docs/analytics/apireference/all Last Access in February 2020 4 Network Simulator 3 (ns-3): https://www.nsnam.org/ Last Access in February 2020 of CPU usage, the percentage of GPU usage (clustered by the used GPU engines), and the RAM usage. When the process is finished, the tool calculates statistics (max, min and average values, per interval if desired) and writes a log file (with the execution time as id) for posterior analysis.

Requirements.
Requirement a) The software needs to be run in Windows devices, because it makes use of the Windows PowerShell. Its previous installation is then required.

Metrics.
The following metrics are measured and registered by making use of the RCM-UV tool: a) CPU usage: The measurement of the CPU usage when performing data-intensive tasks is important to know about the requirements of specific components and the resources that are needed to meet them. The CPU usage is typically measured in percentage. However, the percentage usage (i.e., CPU load in %) is highly influenced by the computational resources of the involved processors/machines. Therefore, the CPU usage in terms of amount of memory (bytes) can be also measured.
b) RAM usage: The same rationale for measuring this metric than for the CPU usage applies. It is measured in amount of memory (bytes). c) GPU usage: The same rationale for measuring this metric than for the CPU usage applies. It is especially relevant when performing video-or graphic-related processing tasks. It can be measured in terms of percentage and bytes, and gives information about the number of engines used.

Implementation Overview.
The RCM-UV tool makes use of the <<System.Management.Automation.dll>> library of the Windows PowerShell to trigger queries for the particular process id (PID) of the indicated process name from which to retrieve the targeted metrics. An example of how to get the PID is shown below: Process[] activeProc = Process.GetProcessesByName("process_name"); int unity_pid = activeProc[0].Id; An example of the queries sent to the Powershell to retrieve the CPU usage is shown below: //CPU usage: query String ps_script_cpu_first= "(Get-WmiObject -class Win32_PerfFormattedData_PerfProc_Process -filter IDProcess="; String ps_script_cpu_second= " | select-object PercentProcessorTime).PercentProcessorTime"; //CPU usage: parsing the obtained values cpu_str = PowerShell.Create() .AddScript(ps_script_cpu_first + unity_pid + ps_script_cpu_second) .AddCommand("Out-String") .Invoke<String>()[0].ToString(); The source code for the complete program lifecycle, for each of the considered metrics, and for their posterior statistical analysis, can be checked at the Github.

Usage.
Step 0. Make sure that the Windows Powershell is successfully installed before running the tool.
Step 1. Run the RCM-UV tool (e.g., double click on the .exe file).
Step 2. Indicate the name of the program to be analyzed. Then, the RCM-UV tool will check if the specific program is already running to start registering the resources consumption metrics. If not, the tool will ask the user to start running it.
Step 3. While the program is running, the tool will periodically measure the metrics -the measurement interval can be configurable -and display the obtained values in the console. When the execution is completed, the tool will calculate global statistics, display them in the console (see Figure 1), and register all compiled data in a log file.
Step 3. The user can check the displayed information in the console and the log files (created within the log folder).
The necessary steps to use the RCM-UV tool are explained with further details in the Readme section at the Github, being accompanied by screen captures and a demo video (https://www.youtube.com/watch?v=yPMkxQbk0fE).

Objectives.
The DQM tool has been developed to be able of measuring and registering QoS and QoE related metrics for DASH streaming sessions, when using either native or web-based players, even those lacking of metrics measurement capabilities. The set of metrics that are measured extends those measured by the state-ofthe-art APIs and players, like e.g. dash.js.

Requirements.
Requirement a) The software needs to be run in Windows devices, because it makes use of the Windows PowerShell. Its previous installation is then required Requirement b) The software makes use of TShark 5 , the terminal oriented version of Wireshark, to monitor and analyze the DASH traffic. Its previous installation is then required, and the path to its installation folder must be provided.
Requirement c) The software is valid for non-secure HTTP connections. The DASH traffic over HTTPS (HTTP Secure) is encrypted, and thus cannot be analyzed (in this current version).
Requirement d) The software requires the capture of the MPD to run successfully. Therefore, the web browser cache must be cleared before running the software. Otherwise, in case that the MPD has been downloaded before and is stored in the cache, the software will not work successfully.

Metrics.
The following metrics are measured and registered by making use of the DQM-UV tool: a) Round Trip Times for DASH chunks: total time since an HTTP GET is sent to the DASH server (e.g., a conventional HTTP server, like Apache), and all associated TCP segments for the associated DASH chunk have been downloaded. b) TCP Segments Count and Length: number of TCP segments downloaded for each DASH chunk and their length in bytes. c) URL of the selected DASH chunk: with this information, the associated parameters from the MPD can be obtained, like the codec, resolution, framerate, bitrate, audio channels, audio frequency, etc. This information is extracted from the MPD, which is also captured and downloaded. d) Effective bandwidth: it is calculated by dividing the total size of each chunk into the time required to fully download it. e) Total number of frames and bytes downloaded: This is relevant to know about the total amount of data downloaded, enabling e.g. a comparison between conditions and players.

Usage.
Step 0. Make sure that the Windows Powershell and TShark are successfully installed, before running the DQM-UV tool.
Step 1. Run the DQM-UV tool (e.g., double click on the .exe file).
Step 2. Confirm or indicate the TShark path (it is read from a text file in the program folder).
Step 3. Select the network interface from which the DASH traffic will be captured. If no interfaces are listed, then execute the <<sc start npcap>> command with admin rights, and try again.
Step 4. Select the time duration for the capture.
Step 5. Select the desired intervals for which the capture needs to be divided. By default, a single interval for the whole period is used.
Step 6. Select whether a single process or multiple process program will be analyzed. The selection of the multiple process option is required for web-based DASH players, as web browsers commonly instantiate new processes and threads during their execution.
Step.6.a) If a single process is selected, then the user must indicate either: Step.6.a) a) The process name. In this case, as the tool needs to wait until the first connections are established, the initial MPD request can be missed, but the subsequent DASH traffic will be captured.
Step.6.a) b) The URL of the MPD to be captured, if known. Then, the validity of the URL will be verified, and the user will be requested to indicate the process name. In this case, the initial MPD request, and thus the MPD itself, will be also captured.
Step.6.b) If the multiple processes option is selected, then the user does not have to indicate anything else. The whole traffic will be captured. Then, the MPD request will be searched for and the subsequent DASH traffic will be tracked and analyzed.
Step 7. The desired DASH players, or applications with the DASH player embedded (e.g., Unity build), are prepared.
Step 8. The capture is started.
Step 9. The player is launched.
Step 10. When the execution is finished, the statistics and captured traffic will be displayed in the console (see Figure 2). In addition, the following files will be created and stored in the logs folder: 1) the MPD; 2) the raw capture with the same info displayed in the console; 3) the info about the downloaded DASH chunks for each audio and video AdaptationSet.
The necessary steps to use the DQM-UV tool are explained with further details in the Readme section of the Github, being accompanied by screen captures and a demo video (https://www.youtube.com/watch?v=rE4-iGGr30A).
Finally, two characteristics of the DQM-UV tool must be highlighted: -It has been developed for a specific format of DASH compliant MPD, validated via: https://conformance.dashif.org/ An example of MPD for which the tool works well is: http://84.88.32.46/mtails/resources/MTAILS/Tears_of_steel_TV/s tream.mpd It might not work for other DASH compliant MPD variants, but the source code of the tool could be easily adapted to properly process the target MPD.
-It has been tested for media assets with a single audio and a single video AdaptationSets. If further AdaptationSets are available and selected by the DASH player under study, then the tool will work well, but just a video and an audio log files will be generated. The source code could be also easily modified to adapt to these new requirements.

Conclusions & Future Work
This paper has presented two valuable open-source software tools for the measurement of resources consumption metrics for Windows applications and of QoS metrics for DASH streaming sessions when using non-secure HTTP connections. Both tools are completely operational and have been released in Github (https://github.com/ETSE-UV/), such that the community can take advantage of their existence, and customize them to the particular needs, if necessary.
These tools are going to be used in ongoing research activities. On the one hand, the RCM-UV tool will be used to evaluate the performance and requirements for different Unity-based and webbased players, when playing out different combinations of traditional and immersive media formats using different devices. On the other hand, the DQM-UV tool will be used to analyze the obtained QoS in different DASH streaming scenarios, considering both traditional and immersive media formats, via different network connections, under different conditions, and when using different media players and ABR strategies.
Apart from these planned tasks, future work will be targeted at extending and refining these tools. First, the tools will be adapted to provide universal support, not just limited to Windows devices. It will be achieved by making use of Linux software tools and dockerizing them. Second, graphical user interfaces and (realtime) metrics visualization and analysis tools will be integrated. Third, an additional tool will be developed to be able to accurately reconstruct the received video at the client side.

ACKNOWLEDGMENTS
This work has been funded by Fundación BBVA with a "Beca Leonardo a Investigadores y Creadores Culturales 2018", and by the Spanish Ministry of Science, Innovation and Universities with a Juan de la Cierva Incorporación grant, with reference IJCI 2017 34611. Additionally, the work has been carried out within a close collaboration with the EU H2020 VR-Together (agreement nº 762111) and ImAc projects (agreement nº 761974).