Lessons Learned from the Quality of Experience (QoE) Assessment of 4G Mobile Technology in Indonesia

Received Dec 14, 2017 Revised Feb 10, 2018 Accepted Mar 21, 2018 The purpose of this research is to assess the quality of experience (QoE) of 4G mobile technology that is used in the Indonesian market. The study analyzed the results obtained and discussed the lesson learned for stakeholders of the telecommunications industry. Primary data was obtained by conducting a market survey in May 2017, with a confidence level of 95%, representing the 4G subscribers in the Indonesia market. There are three main subjective aspects of QoE: service quality, data speed, and network quality (indicated by stability of signal). The analyses were conducted based on ACR-HR score, comparing users’ experiences of 4G with their previous 3G subscriptions. It was found that, in general, the Indonesian market perceived 4G as offering a better experience than 3G. However, one of the important lessons that can be learned is that network operators should provide a more stable signal, as the market gave a lower score for this aspect.


INTRODUCTION
The International Telecommunication Union (ITU) has defined quality of experience (QoE) as the "degree of delight or annoyance of the user of an application or service" [1]. In contrast to quality of service (QoS) assessment, QoE assessment is mainly based on user habits (i.e. it is user centered), while QoS assessment is based on technical performance (technology centered) [2]. In the context of mobile communications networks, QoE assesses the users' perceptions of the service, and these perceptions may relate to all aspects of the technical system, including terminals, networks, infrastructure, etc. Hence, QoE assessment is essential, since experience gained from previous services, as well as expectations for the new service, influence future strategic plans. For example, QoE assessment data can be used to overcome the problem of churn rates and the exclusion of new service plans [2], aspects of strategic capability that cannot be satisfied merely by utilizing QoS data [3].
Indonesia, the fourth most populous country in the world, commenced its commercial implementation of 4G mobile technology in 2015. However, public QoE assessment of 4G has not been officially reported since its commencement, despite the fact that QoE data is essential for supporting the continuing strategic plans of mobile network operators. Lessons that can be drawn from current market experience are no less important for the success of future technology rollouts than they are for the current 4G service.
This research aims to assess the quality of QoE of 4G mobile technology in the Indonesian market. Subsequently, lessons learned for the mobile network operators are derived. A market survey was conducted in May 2017, with a minimum sample complying with Slovin's formula, giving random sampling with a confidence level of 95%. The respondents were requested to give their subjective perception of 24 items relating to three assessment aspects, these being service quality, data speed, and stability of signal for the 4G network. To provide a comprehensive analysis, the ACR-HR method was used to analyze the survey data, represented by scores ranging from -5 (poorest) to +5 (best). Such a scoring system provides a comparison between users' current experience of 4G services and their previous experience of 3G. The research theme presented in this paper is part of ongoing work investigating the shifting of the Indonesian market from 3G to 4G [4], [5].
The remainder of the paper is structured by reviewing underlying theories in Section 2 and presenting the method of market survey undertaken in Section 3. The results and discussion of lessons learned are presented in Section 4 and the conclusion is presented in Section 5.

THEORY OF QoE ASSESSMENT
The definition of QoE given by the European Telecommunication Standards Institute is "a measure of user performance based on both objective and subjective psychological measures of using ICT service or product" [2]. In line with this, Recommendation ITU-T G.1011 sets out two overall methods to be used for assessing QoE: subjective assessment through perception surveys, and objective assessment through monitoring, planning, and/or lab testing.

Objective Assessment
Objective assessment of QoE involves the use of an estimation model matched to the actual technological service or product being reviewed. The model should comply with the service or product objectives, provide the new application features, and have the new network configurations. In regard to communications technology, objective QoE assessment can be carried out in three ways: intrusive mode, non-intrusive mode, and planning mode [6]. In intrusive mode, a signal is injected into the system under test, with the aim of measuring the output signal. Non-intrusive mode assessments measure traffic directly without adding a signal to the system, while assessment in planning mode is not performed in a real system and usually serves as a tool for designing future network systems. Such a mode requires a planning-specific assumption as well as detailed knowledge about the service or product being assessed [6]. Objective assessment is cheaper and faster than subjective assessment, but the accuracy of the results obtained depends on the suitability of the model. Hence, objective assessment is considered to perform with limited validity and reliability. Similar to QoS assessment [7], such an objective assessment would focus on the QoE technical metrics of a service application. The example of objective QoE assessment would be a laboratorybased assessment of a new video service before it is commercially launched to the market [8] and the PSNR evaluation of video application in wireless multimedia sensor network [9]. In relation to the 4G network, the Office of Communication (OFCOM) in the UK conducted a data speed test and reported that the UK 4G network performed at a faster data speed than the 3G network [10].

Subjective Assessment
This research is developed based on subjective assessment. Subjective assessment of QoE measures experience, perception, and satisfaction as they are defined by users. Users are asked to provide opinions about the quality of various aspects of a service or technology, expressed using a rating scale known as a mean opinion score, or MOS. ITU defines MOS as "the value on a predefined scale that a subject assigns to his opinion of the performance of the telephone transmission system used either for conversation or for listening to spoken material" [11]. Users respond to questions about their levels of satisfaction with service operations through survey techniques such as interviews, discussion groups, and questionnaires. In comparison with objective methods, subjective QoE assessment requires higher budgets and more extensive resources. Subjective assessment of QoE can also measure overall system performance by taking into account technical parameters (QoS). Hence, QoE data can combine knowledge of the users' subjective experiences with the system's technical parameters. An example of this method is the QoE measurement of the 3G network in the UK [12] conducted by OFCOM, one important finding of which was that users were most concerned about the availability of voice-call services and clarity of sound. Another example of the QoE subjective assessment includes the evaluation of users' perception of computer games application [13].

METHOD 3.1. Primary Data of QoE Subjective Assessment
A survey is a much-used tool for obtaining primary data for subjective QoE assessment. For this study, a market survey was conducted in May 2017, questioning respondents using a random face-to-face method. The number of respondents in the sample complies within Slovin's formula to represent a greater than 1 million population with a margin of error of 5%. The main requirements for the chosen respondents were as follows: a. Already be subscribed to 4G mobile service (either as prepaid or postpaid users) for a minimum of one day. b. Be resident in Jakarta (the capital city of Indonesia), with his/her main routine activities being located in the city. c. Have subscribed to a 3G service before using 4G technology. d. Use a mobile handset as the terminal to access 4G service (in some countries, including Indonesia, 4G is also used for fixed-data communication services). The sample area was Jakarta, a city with a population of around 12 million. Previous research carried out by the authors has argued that Jakarta can be used as a sampling area representing the national market because [4]: a. Jakarta is the capital city, the 'life style' center of the country, the most multicultural city in Indonesia, and the center for modern trends that are subsequently followed by other parts of the country. b. Typically, the national operators launch their latest services, including 4G, in Jakarta, and penetrate into other service areas afterwards. c. Despite the heterogeneity of the country's demographic profile, around 66% of the population of Jakarta is Javanese. This proportion is similar to the overall distribution of population in Indonesia. d. Some 60% of the Indonesian money supply is located in Jakarta, 30% in other cities and 10% in rural areas. Therefore, all mobile network operators have treated mobile subscribers in Jakarta as their main revenue sources, taking into account both their dynamic and routine activities (the term 'money supply' refers to the amount of money available in an economy).

Questionnaire Design
Factors that influence QoE are any characteristic of a user, system, service, application, or context whose actual state or setting may have an influence on QoE for the user [3]. In this research, mobile users were requested to respond to 24 statements grouped into three assessment aspects: service quality, data speed, and stability of signal. These assessment aspects represent factors influencing QoE. For each statement, respondents were asked to give a score representing their subjective evaluations of 4G and 3G mobile subscriptions. The scale used was an absolute category rating (ACR) of 5 points, graded as follows: 5=very good, 4=good, 3=normal/ordinary, 2=poor, and 1=very poor. Table 1 gives a selection of the statements included in the questionnaire. In the questionnaire, respondents are requested to express what they perceive individually for both 3G and 4G service. Using the respondents' scores, absolute category rating with hidden reference (ACR-HR) was calculated by subtracting the 3G scores from the 4G scores. The difference between the MOS value of the 4G service and the 3G service is an adaptation of the Recommendation ITU-T P.910. ITU defines the ACR-HR method as a category judgment where test sequences are presented one at a time and are rated independently on a category scale [14]. Since the main QoE assessment being sought is for 4G the reference contained in the procedure is 3G service.
The ACR-HR scale used in this research to translate the qualitative comparison values is as follows: a. ACR-HR range from -5 to -2: 4G service is perceived as much worse than 3G b. ACR-HR range from -2 to -1: 4G service is perceived as worse than 3G c. ACR-HR equal to 0: 4G service is perceived as just the same as 3G d. ACR-HR range from 1 to 2: 4G service is perceived as better than 3G e. ACR-HR range from 2 to 5: 4G service is perceived as much better than 3G Table 2 presents the basic data for the respondents. It shows that the majority of respondents were male. The age of respondents closely follows a normal distribution, and sampling is predominantly of those aged 19 and 20. The data obtained shows that 86% of respondents had used 4G for more than three months while 71.09% of respondents used the same mobile phone brands for both 3G and 4G services.

1207
The survey found that students and private company employees (including state-owned companies and foreign employees) constituted the majority of 4G service respondents by profession. This suggests that students are the respondents who most easily accept new technology. In terms of network operators, the results mirrored the actual market of six main operators, in which Telkomsel is a dominant national operator with a market share of more than one third.
The basic data also shows the locations most frequently used to access 4G services. The most commonly used location for accessing 4G was at home. Mobile phones are telecommunication technology innovations developed for mobile use but are currently used to access information and telecommunications most often when mobile phone users are at home. Figure 1 lists details of service quality aspects with their ACR-HR score results. Overall, it can be seen that none of the scores exceed 1.5. The results signify that 4G service qualities are perceived as "better" than 3G, but with the lowest ACR-HR scores given for traditional voice-call sound quality and SMS service quality. These results are not surprising since voice calls in the Indonesian mobile network are still carried on the 2G network, 4G-based voice technology (such as VoLTE) having not yet been implemented. The current voice-call platform is circuit-switched (CS) fallback, which enables the provision of voice and traditional circuit-switched services. Figure 1 also indicates that users appreciate the significant service improvements provided by 4G for video calls and multimedia streaming (such as YouTube, JOOX, Spotify, etc.) suggesting that significant technical advance has provided a better experience for 4G subscribers. Additionally, the Indonesian market also appreciates the improvement of the sound quality for voice calls using messenger applications. However, the average score for this aspect is quite low (1.0 out of 5.0), indicating a small level of improved QoE when compared with 3G.  Figure 2 provides details of the data speed aspects and their ACR-HR score results. Overall, it can be seen that the scores range from 1.0 to 1.5. All the 4G data speed aspects are perceived as "better" than 3G data speed, while no single experience is perceived as "worse". The maximum ACR-HR scores are given for the experience of uploading files, downloading files and Wi-Fi tethering (using handsets as personal hotspots). The results indicate that current implementation has fulfilled the basic purpose of 4G technology, which is to enhance data communications over previous mobile technology evolutions. Technically, 4G should accomplish data communications of up to 1 Gbps in stationary states and 100 Mbps in moving conditions. However, we cannot judge the current subjective QoE as inferior, because the results indicate a score of "better", although the score for "better" is only just beyond the "same as before" range. In fact, most of the Indonesian market has enjoyed 3.5G technology (such as high-speed packet access, or HSPA), which can, in some cases, give data speed of hundreds of Mbps. Figure 3 provides details of the network quality aspects and their ACR-HR score results. Overall, it can be seen that the scores do not exceed 1.2. The subjective indication of network quality is represented by the term "stability of signal". Mobile subscribers can easily judge this based on technical indicators in their handsets. The results show that the stability of 4G signals is perceived as "better" than 3G services. However, the network quality for traditional voice calls has a score of less than 1.0. Such a near-to-zero score indicates that users' experience of 4G is almost the same as their experience of 3G. This is consistent with the QoE results for the service quality aspect, since the voice-call network is still running on the 2G network (using the underlying CS fallback mechanism). Another interesting finding is that the Indonesian market regards data service as being more stable than voice service. This is, therefore, a reliable indicator for mobile network operators to consider the implementation of voice communications services, such as VoLTE, which are fully delivered over the 4G data network.  Figure 3. QoE results for network quality aspects

LESSONS LEARNED
In the final part of the questionnaire, the respondents were asked to give general QoE conclusions for their current 4G subscription. The results indicated that 59.48% of respondents perceived 4G service as "better" than 3G, while 34.36% of respondents perceived 4G service is "much better" than their previous 3G service. Hence, Indonesian operators can be seen to have delivered significant performance improvements for 4G services, in that users notice positive differences and perceive them as being better than 3G. For each assessment point investigated, the data speed and network quality aspects provided positive ACR-HR scores. Such data should be easily understood and translated into strategic references for mobile network operators. QoE data is also useful for optimizing products and/or services [2], because technical teams may not have had sufficient knowledge about the specific technical parameters that affect user experience. Some of the main points that can be further elaborated as strategic references are as follows: a. Operators should pay special attention to the aspect of network quality, since stability of signal receives a lower ACR-HR score. This survey result may also indicate that subscribers have experienced some blank spots as well as unstable coverage. One possible cause would be that, as the development of 4G started only a few years ago, 4G coverage is not as extensive as 3G coverage. It is therefore suggested that Indonesian operators should accelerate infrastructure rollout. b. QoE data is also useful for optimizing and improving current data services. Technical teams often focus on the product, but do not have much knowledge about the specific technical parameters that affect enduser experience. It is perhaps believed that user experience is a subjective measure, but it can also be analyzed objectively. In other words, QoE, which reflects the measurement of user's psychological feelings, should also be expressed in terms of QoS characteristics. In this way, data for QoE can be combined with technical QoS parameters, providing valuable information about user experience. For example, this QoE assessment has indicated that most subscribers are actively using messenger applications such as WhatsApp for voice-call communications. Any poorer technical performance, such as longer delays and lower power signals, will cause a degraded sound quality that is subjectively perceived as inconvenient by users. Hence, technical quality enhancements are needed to satisfy users' subjective experiences if voice communications services are to be preferred over messenger applications. c. QoE data can help in overcoming the problem of high churn rates. Profits and brand image will be negatively affected if service consistently fails to meet consumer expectations. Opportunities to develop customers' loyalty will be increased by developing improvements in current services with the help of QoE data. For example, the survey shows that customers value data speed for mobile tethering (using mobile handset as Wi-Fi hotspots). Hence, operators should pay attention to such aspects of market behavior in Indonesia, where consistent development of the relevant service will be likely to increase brand loyalty. On the other hand, a chain of negative reactions may easily occur when the mobile network operators, as well as service developers, are unaware of such phenomena. d. QoE can be used as a reference for future service applications. Future services will be adopted by the market, or on the other hand, rejected by the market. One reason for rejection is that consumers can experience a lower service experience when they start to utilize a new service. Such conditions may occur even if the service has received positive feedback during market testing and/or precommercialization advertising. Such a rejection can be predicted and avoided if QoE data is analyzed accurately and in detail through user testing before the product is launched. e. QoE assessment has also shown that subscribers appreciate faster data, suggesting that operators should prioritizing relevant data communication applications. However, mobile network operators need to survive in the era of OTT (over-the-top) content. This presents a challenge, as mobile operators have had to invest to build infrastructure, but do not enjoy significant revenue from the commercialization of applications, because subscribers are paying for the data network, not the service applications.

CONCLUSION
This paper has presented QoE assessment results for the 4G mobile technology that has been rolled out in the Indonesian market. A market survey has been conducted with a confidence level of 95%, investigating the 4G subscriber in the Indonesia market. Three main subjective aspects of QoE assessment have been investigated: service quality, data speed and network quality (indicated by stability of signal). The analyses were conducted based on ACR-HR score, comparing users' experiences between 4G and their previous 3G subscriptions. For the service quality aspect, the ACR-HR scores were positive but never . This result signifies that 4G service qualities are perceived as "better" than 3G, but with the lowest ratings in this aspect for traditional voice calls and SMS. For the data speed aspect, the scores ranged from 1.0 to 1.5, indicating the services are perceived as "better" than 3G data speed. In this aspect, the maximum ACR-HR scores were given for the experience of uploading files, downloading files and Wi-Fi tethering (using handsets as personal hotspots). For the network quality aspect, the QoE assessment had no scores that exceed 1.2, with the network quality of traditional voice calls having a score of below 1.0. Such a near-to-zero score indicates that the current experience for making traditional voice calls is felt to be no different to the previous 3G service. To conclude, based on QoE assessment results for these three aspects of user experience, the Indonesian market perceives 4G as providing a better experience than 3G service. The QoE assessment results can be further developed into strategic lessons for mobile network operators. From these five lessons, a particularly important issue highlighted is that Indonesian operators should pay special attention to the aspect of network quality, since the ACR-HR score for stability of signal is low.