The End User Requirement for Project Management Software Accuracy

ABSTRACT


INTRODUCTION
Applying the software as a PMS (Project Management Software) in a new project system is intended to manage the high quality of the project information of all project organizations. It is not like turning the palm of the hand; At least, there is the difference in the cost platform with the current systems. This study is based on the difference in cost output of PMS toward the current method, namely SNI (Indonesia National Standard), on projects in Aceh-Indonesia. These different raise a presumption about using the inaccuracy of PMS that is implemented to the projects in Aceh-Indonesia. Both the PMS accuracy and the end user requirement problem cannot be generalized, each depends on the case by case on the project standard used and the end user group domain, especially in Aceh-Indonesia who use SNI for project standards. It is possible to so because the implementation of the PMS on other platforms and the end user will give different results from this research. However, the method that will be presented in this research can be applied to another platform and the end user domain. Some of the studies have been conducted that relates to the impacts of PMS in organizations concerning to its productivity, the process, and innovation [1][2][3][4]. Unfortunately, the research has not shown how the impact could reach the end user requirements. This research will be conducted to fill the gap between PMS accuracy and the end user requirement. The PMS testing should be conducted earlier before the PMS is really applied in a project. The testing must meet the PMS functionality characteristics, which one of them is characteristic accuracy [5].
The end user, who has the authority to specify the requirements for the use of the PMS, should conduct the test and provide suggestion to the PMS producer for accuracy improvement. It will give benefit and satisfaction to the end user itself and the PMS producer. There is no an exception to the PMS that should  (Fachrurrazi) 1113 also have to meet the requirement of the end user in Aceh. Following this conception, the customer is allowed to express his real needs, which may differ from the PMS specifications on one or more project issues. One of the important function of the PMS is the controlling tool for the project cost as introduced in the paper of [6] and it can cause the need for software accuracy is also getting higher. The PMS with a low accuracy should be tailored to the needs of the end user along with the implementation of the PMS itself. This is one of the problems that should be solved in the practice of project implementation. This problem that is considered as unbalanced and error, was occurred in some projects in Aceh. This difference indicates inaccuracy of the PMS according to the end user. Whether this indication could be directly deduced by the end user as it has been previously expressed, of course, it still requires further verification. The hypothesis testing will be conducted to get the proving for PMS accuracy and we also measuring the probability of the PMS accuracy to achieve the end user requirement. Based on this background, the aims of this research are to analyze the PMS accuracy and to measure the probability of PMS accuracy in achieving ±1% of the end user requirement. To analyze the accuracy, it requires the true value as a proxy to measure the bias performance of PMS (the difference between the observed and the expected/target). The biases%, as a general parameter, could be used to determine the accuracy of the PMS. Meanwhile, the zero value of bias% could be used as an indicator/benchmarks of performance achievement of an application [7]. Some researchers have suggested that the review of the project system is very important [6]. Some of them are related to the satisfaction of the end user for using the PMS and reaching the project administration goal. In developing countries, the use of the certain PMS, like Primavera, Microsoft Project, etc., is usually implemented without conduct the test first. This will cause a bigger problem when the PMS has been implemented in managing the project running.
The methods to control the PMS fault or errors, likes fault tolerance method [8], statistical techniques based on the error [9], are the challenges faced by the software industries today. In this research, the method that is used to assess the PMS accuracy is statistical techniques with a hypothesis and normal distribution. The null hypothesis statement is that the PMS is still accurate for its use in projects using the SNI platform. The decision to reject or no reason to reject the null hypothesis is formulated using P-value approach compared with α (for two tail) in a normal distribution case. The parameter in this hypothesis is the average of bias% as its central tendency. In case of fail to reject the null hypothesis (or the PMS still accurate for the project in Aceh), we will analyze the PMS accuracy using error tolerance and the next we will analyze the probability to achieve the end user requirement using the normal distribution density curve. To conduct this research, we have collected 37 data from projects that using PMS in Aceh-Indonesia.
This paper focuses on analyzing the PMS accuracy and its relation to the end user requirement, which is the starting point for the project management performance. In this study also has been done a gap analysis between the PMS accuracy and the end user requirement, by explaining specifically regarding the probability that can be achieved by the PMS to meet with the expectations of the end user. This is a novelty in this research study, in which some previous studies have only examined one of them, between the PMS accuracy or the end user requirements without reconciling these two conditions. This study has developed a procedure to achieve the intent.
Operationally in this research, we have been able to reject the end user presumption that the PMS is not accurate through hypothesis testing. Conducting test how accuracy level of PMS can fulfill the end users requirement that it does not exceed ± 1% error by measuring the probability of ±21.77%. In this paper, we also have been able to formulate predictor analysis to generate the PMS cost from the SNI cost (current method) by presenting the PMS accuracy index of ±7.5%. To increase confidence in this study, we also have assessed the closeness between the PMS (observed) and SNI (expected) by correlation analysis with R 2 of 0.995. This is done to show that the data used by both the PMS and the SNI derived from the same source.

LITERATURE REVIEW 2.1. Accuracy
There are many different understandings about the notion of accuracy. This is natural if the notion is in a different context. In the context of the software accuracy according to [10], accuracy has been defined for individual measurements as the sum of the absolute value of the systematic error plus the standard deviation of the measurement [10]. ISO defines accuracy is a combination of both types of random and systematic as a measure of statistical bias [11]. Generally, the accuracy system is related to systematic error and random error. They are two types of error that have a difference in characteristic of the error source. Many researchers are confusing about the two type of the error; it is caused by the differences in background. The real error is the sum of the systematic error and the random error.

Random and Systematic Error
Random errors arise from temporary factors and their fluctuations are unpredictable. It has an error contribution that affects the precision of a test result. Examples of random errors are such as the effect of electric voltage fluctuations, temperature, humidity and other condition of experiment environments. Criteria for random error are often implemented by using the normal distribution, using the tolerance term to accompany the accuracy, using the concept of the mean and standard deviation as the best estimate/central tendency, the error bound could be reduced by averaging over a large number of observations, etc.
Systematic errors are the bias that occurs regarding a system and usually expressed by a mathematical formulation [12]. Any change in one or more of the system elements will cause a change in the character of the systematic effects. Criteria for systematic error are using the zero bias concept as the true value, arising from defects in the instrumentation, using the precision term to accompany the accuracy, and having the same error in repeated experiment/observation, etc.
Implementation of accuracy uses the tolerance analysis is often used in industry/manufacturing. This method is used to predict the probability of a production defect as acceptable of the production machine [27]. The observed data could be hard to interpret properly if we do not know the object measured [28].

Research Framework
The research framework has been built to conduct the research, as shown in Figure 1. This framework explains the hypothesis testing of the difference between two population means (bias% and true value). If the bias% is not a significant difference from the true value (zero bias%) then the PMS (as observed) is still accurate. If the bias% is a significant difference from the true value (zero bias%) then the PMS (as observed) is not accurate. This research use the bias measurement concept that it is understood in terms of systematic error and random error, it is in line with the research [29]. To perform this test, we first need to know whether the sampling bias% has a normal distribution or not. The purpose of this normality test is to determine the proper technique to use on the hypothesis testing (namely: parametric statistic or nonparametric statistic). The bias% data is calculated using formula 1 and formula 2. The result of the bias% as shown in column 5 in Table 1 (attach in the appendix).

Hypothesis
A hypothesis test is the evaluation of the two statements for both the null hypothesis (Ho) and the alternative hypothesis (H1). It is regarding the mutually exclusive of the population in determining the best statement that is supported by the data sample. The null hypothesis is mean of bias% is equal to 0% (zero percent). This test is performed with the confidence level of 95% or interval confidence α±5%. The bias%, as the hypothesis variable, is calculated by using formula (1), (2). the hypothesis formula for the following: Null Hypothesis, Ho: µbias% = 0% (as the true value of µbias%) Hypothesis alternative, Hi: µbias% ≠ 0% (as the true value of µbias%) The setting up of rule for decision in reject or fail to reject the null hypothesis is based on P-value (calculated probability). We will reject the Ho (null hypothesis) if P-value less or more than significant level of α±5%. Otherwise, it fails to reject Ho if P-value is equal than significant level of α±5%.

RESULT 4.1. The Bias% Analysis
A number of the data as much as 37 have been collected to review the accuracy of the PMS, as shown in Table 1. Bias% is displaying the percentage of the difference between observed and expected compared to expected, as defined in formula (2). If the observed have the same value as the expected, then it is said to be an unbiased or zero bias%. The unbiased condition is rare in projects, it is due to the systematic error of PMS and the random error, ie. the number of project activities, the decimal rounding, truncation, human error, etc. Here we presented the results of the calculation for the bias% of the PMS as a basis for analysis levels of accuracy of the PMS with the formulation of bias% are: Observed ( ) is the total cost of the PMS output. Expected ( ) is of the total cost of the current method output. Bias% is the percentage of bias of observed dataset toward expected dataset (current method). Based on the calculation of the bias% in each project, as shown in Table 1 (attach in appendix), shows the values vary between -/+. It indicates that the PMS has an error distance to a true value (0%) in -/+ variation. We have a question in this indication, is there a pattern that could be drawn from increasing/decreasing for the PMS output (observed) toward its bias%? The result of the correlation test, with R-squared is -0.01117 (no correlation). We conclude that there is no pattern between increasing/decreasing of the PMS output in line with the increasing/decreasing of its error (bias%). This is a probability condition of the PMS output. This possibility could be due to variation on the the number activity of each project (random error), or due to rounding and truncation (systematic error). The average of bias% of the PMS is 0.4066% with standard deviation of 3.6182%, as shown in Table 1. This value will be used to build a normal distribution.

Chi-square Test and Normal Distribution
The requirement of the statistic parametric is the normal distribution of data, which will be tested by using chi-square. This is also called the normality test or the goodness of fit. The chi-square test is to calculate the probability of the differences between two data sets (frequencies of observed and expected) using formulation: Where is the observed; is the expected; The expected are obtained from a function of normal distribution. Chi-Square analysis (χ 2 ) is sum total of squared error in each class (for observed and the expected). The result analysis of chi-Square test (χ 2 ), as shown in Table 2. The result of the chi-square of 27.51 is the lower than the chi-square critic of 51. We conclude that bias% data are distributed normally. Based on this result, we can use statistic parametric. It is in line with [30] and state that, the parametric test is based on a) the variables must be expressed in interval scale of measurement; b) the population values are normally distributed; c) the subjects are selected independently; d) the samples have equal or nearly equal variances. To conduct hypothesis testing, the ANOVA approach will be applied to this research. The normal distribution is universal and could be used to measure the central tendency of the mean of bias% (µbias%). The normal distribution curve has been built based on µbias% of 0.4066% and standard deviation of 3.6182%, as shown in Figure 2. This curve shows the gap between the true value of bias% (zero bias%) and the estimated mean of bias% (µbias%), as shown in Figure 2. Furthermore, we examine the significant differences of the true value (0%) and the mean of the bias% (0.4066%). To examine this hypothesis test, we will use Anova.

Analysis of Variance (Anova)
ANOVA at the confidence level of 95% will be used to analyze P-value. P-value will be compared to α(5%) for the decision to reject or fail to reject the null hypothesis (Ho). The P-value will be used as the  (Fachrurrazi) 1117 parameter of the testing for all levels of significance that it may be achieved. The P-value ≤ α5%, the null hypothesis (Ho) is rejected, and vice versa it will fail to reject it. The P value of 49.6436%, as shown in Table 3, is bigger than α (interval confidence) of 5%. The conclusion of this result is to fail to reject the null hypothesis (Ho). This is given the observed is not a significant difference in the expected value (true value). In this result, we conclude the PMS is still accurate to be used in projects using SNI (National Indonesia Standard Platform). The second problem in this research, whether the bias% will provide a high value to the expectations probability for ±1% of the end user requirement? as indicated by the normal distribution curve in Figure 2.

The Probability of the PMS to Obtain the End User Requirement
In this research, the end user expects the difference between the outputs of the PMS to the true value is not more than ± 1%. This is the tolerance number determined by the end user against errors in the PMS. As, some the end users have reported errors of the use of the PMS, containing one or more mistakes, will not exceed ±1% [31].
Base on analysis on Table 4, the probability expectation of PMS is achieved 21.77% of the end user requirement (not exceed ± 1% error). Although the level accuracy of PMS is accurate, this probability value is low. This is important to fulfill the requirements of the end user, where the PMS credibility will increase if the PMS could be met the needs of the end user requirement [32]. The accuracy requirements of the end user and its probability, as shown in Table 4, are resulted from normal distribution area as shown in Figure 2.

Interpretation of the PMS Accuracy
The accuracy analysis of PMS is related to an error in the project activities. The difference number for each project activity will cause variance in error of each project that differences one to another project (P1 to P37). We use the tolerance (It is the permissible limit or limits of accuracy) to measure the accuracy level of the PMS that refers to the interval of variability in a normal distribution. The concept of measuring accuracy by using the tolerance (using interval level) of the PMS is fairer than using only the discrete average (mean) of bias% as accuracy. It is based on the suitability attribute of the PMS's ability to meet the probabilities of the end user requirements (which are related to the extent of the normal distribution). This condition illustrates, the greater the variation of bias% will have implications on the greater the tolerance of observed (a measurement object). The greater the tolerance of measurement results will have implications for the lower the accuracy of the observed (the PMS). Based on this implication, we conclude that variations in a number of project activity and the bias in each project activity will have an implication on the accuracy of the PMS, as well as its tolerance. The value of the tolerance could also be categorized as the level of accuracy of the PMS. Therefore, if we want to compare the accuracy level of the PMS to another, then we could compare the tolerance of the PMS with the same number and sample type requirements, which are also obtained from the level of 95% significance. The bias% of PMS in this study is depicted in the normal distribution curve, as shown in Figure 2. The data of bias% of the PMS show distribution is wide, which also describes the tolerance value of the observed. It is in line with [9], state that the tolerance value is expressed in the limit of bias% value and while the interval is expressed as the maximum range of observed data.
Both information on tolerance and interval were analyzed based on the extreme value in the significance level of 95%. Based on the normal distribution curve, the interval of observed is between -6.6849% until 7.4981%, as shown in Figure 3. With this information, we could express the tolerance of the PMS accuracy that based on the extreme value is about ±7.4981% (~ ±7.5%), as shown in Figure 3. Based on the tolerance of the PMS accuracy, we can predict the interval of the observed: (4) Where is the PMS (observed) that will be predicted; is the SNI (expected/targeted). For example P19 data, here will be predicted the value of observed (the PMS) based on target data (value of current methods). We have a SNI cost data (current method) of 2,033,352,413. Using the formula 4, the prediction result the PMS cost output is tolerance range of 1,880,850,982 to 2,185,853,844 (it is obtained from 2,033,352,413 ± 7.5% * 2,033,352,413).
Accuracy index (the tolerance index) in this research is ±7.5% and some research references are called as a value of accuracy [9]. It is commonly used in the context of engineering measurement, which uses a high degree of accuracy and very rarely uses precision terms such as cost estimating [28]. It is often used in a statistical context that uses the concept of variability or random error. It is clear that the type of accuracy measurements may differ depending on the context of the problem and the degree of accuracy. In the cases do not take into account error variability (such as cases of systematic errors and on equipment requiring high accuracy), will have implications for accuracy analysis that also do not take into account precision parameters. Where the precision value is understood as a permissible limit.

The End User Requirement for Accuracy of the PMS
Here we will explain the correlation between the level of accuracy of this the PMS to the end user requirements. As variations of the PMS accuracy, the end user requirements could also vary and depending on the end user group. In this research has obtained the PMS accuracy index is 7.5% and the probability expectation of PMS is achieved 21.77% (it is based on the end user requirement that is not exceeding of ± 1% error). The difference between the PMS accuracy and the end user requirement is not meant that the PMS could not be used to the end users, as its requirement. Only, here the PMS could not meet the expectation of the end user of 100%. In this study the PMS is only able to meet the probability of the end user requirement  (Fachrurrazi) 1119 is up to 21.77%, as shown in Table 4. The end user requirement is the tolerance to the bias of the PMS that the end user expects. We conclude that the greater the tolerance of the PMS, the smaller the probability of the end-user requirements, and vice versa. As the implication of the conclusion is in order to increase the probability of reaching the end user requirement (not exceed ± 1% error) then the variability of the observed (the PMS) should also be reduced. Generally, the requirement error of end user only specifies limitation error requirement, without specifying its probability, so it creates a different interpretation between the end user and producer. The end users should mention in more detail with respect to the minimum probability of acceptance of the PMS accuracy (not exceed ± 1% error). To increase the probability the end user requirement, then the PMS needs to be improved performance accuracy. Performance could be improved if we could find the source of the bias in the PMS system. This is our next research and for other researchers in future.

Validation of PMS Accuracy using Correlation Test
Another way to prove the result of a hypothesis is to review the correlation between the observed and the expected. In this subsection, we will perform a closeness between the PMS and current method. A similar method has been done by [33] with measuring systematic error with curve fits. The closeness between data and trend line is indicated by an R-squared in correlation is less than 1.0 or exactly of 0.995, as shown in Figure 4. This indicates that the error (bias) is very small. This reinforces that the conclusion of the hypothesis states that there is no reason to reject the null hypothesis.

CONCLUSIONS
The accuracy analysis has been performed to assess the cost output of the PMS against current methods. This analysis is performed due to the end user have found there is a difference of PMS cost output from the SNI which is used as a control tool, against the PMS. To overcome the error in performing data input, then the calculation process has been re-examination and recalculation, so the error due to human error could be controlled. For further testing, we have collected a number of data that will be used to test the hypothesis of the PMS output as a basis for assessing the accuracy of the PMS. Hypothesis testing shows no significant reason to reject the null hypothesis. Under these conditions, we have concluded that the PMS is still accurate to be implemented on projects in the Aceh region.
In this study, we also tested how accurate the level of PMS can provide the fulfillment of the requirements of end users. Based on the requirements set by the end user is not exceeding ± 1% error, then the results of this study indicate that the PMS will only be able to provide the probability of achieving the end user requirements is 21.77%. This value indicates the possibility to meet the end user requirements is low. To improve this condition, further research is needed to find the cause of PMS bias in order to minimize the bias% or to increase the probability of achieving the requirements of the end user.
The error could occur while PMS applied to analyze the project costs. however, the project total cost using the PMS can be predicted by using the accuracy index and the SNI project cost. This prediction can be generated by using the formula , Where is the PMS (observed) that will be predicted; is the SNI (expected/targeted); and ±7.5% is the accuracy index of the PMS. In the case of measurement, the accuracy index is also called as the fault tolerance level.
Based on data that has been presented in this research, that is data from project cost from the PMS and SNI, then we can also carry out correlation analysis to data from the PMS cost and SNI cost as a way to