Effective Sampling-Based Assessment Method for Evaluating Electrical Engineering Programme Performance

Received Oct 9, 2017 Revised Dec 20, 2017 Accepted Jan 11, 2018 Challenges in the conventional way of measuring programme outcome include large data sets to be processed, a large amount of manpower, operating cost increment and time consuming. A sampling-based approach to evaluate the performance of a programme is proposed and a mathematical model is developed to determine the overall programme outcome score. The criteria for course selection and percentage of students considered in the assessment, was designed to accelerate the process. The results obtained from the implementation of sampling based approach in the assessment have shown that the programme is successful and the students have achieved the required knowledge and skills. The sampling-based approach is capable of demonstrating the programme and the students’ achievement.


INTRODUCTION
In the implementation of education system that focuses on achieving the specified outcomes, the quality of a programme has to be monitored continuously. Assessment of programme performance is crucial in the monitoring process to identify the required improvement and to provide solutions for several issues [1][2][3]. The solutions can be designed once the analysis of root cause for the issues is determined from brainstorming and research.
There are various ways of evaluating the achievement of a programme. Since Outcome Based Education (OBE) was introduced, the programme outcome measurement was used and a set of performance criteria was established to decide whether the programme has achieved its goal. Direct measurements, indirect measurements or both ways were employed in every course to produce the score for each programme outcome [4][5].
In the programme assessment, measurement of programme outcomes (PO) was carried out from various courses and each course contributes data from various activities which results in large data to be processed. Moreover, large amount of man power is required and it takes a long time to achieve the results, which will increase the operating cost if the measurement is carried out offline and manually. The programme assessment can be done efficiently using a proper technique. It is believed that a sampling based approach of measuring the PO with the online system as the mechanism, may help to accelerate the programme assessment process. Numerous reports have described the outcomes of the course and programme assessments. Most report focused on the types of assessments and techniques used in measuring the course outcomes and programme outcomes at the course level [7,8]. There is no report detailed out the mathematical model used in the determination of the overall PO at the programme level. Some reports revealed that averaging of PO score was used in the final decision of the program attainment however, it is not clear that the types of averaging technique employed in the work [9]. This paper describes the programme assessment process and the sampling based method used in evaluating the performance of Electrical Engineering Degree Programme. In this approach, only selected number of students and courses were considered in the measurement. A mathematical model for determining the final PO scores was developed and employed in the programme assessment. The effectiveness of the method was examined by implementing it in assessing the PO of the Electrical Engineering Degree Programme.

RESEARCH METHOD
The programme assessment process used in the faculty includes a few stages such as performance indicator design, performance assessment matrix preparation, data entry, PO measurement and analysis, Completion of CQI report, PO attainment analysis & Reporting, Approval of the Proposed Actions, Monitoring and Improvement as shown in Figure 1. All level of academic staff starting from lecturer to the faculty management took part in the assessment process.
At the faculty level, the performance indicator for each programme outcome was designed and discussed in the workshop before the programme is assessed. There are twelve POs for each programme as listed in Table 1. The performance assessment matrix was prepared and discussed at the Centre for Studies level. It describes the selected courses for each PO measurement, method of assessment, name of the person in charge and others [10]. The PO measurement was carried out by the course coordinator whereas the PO attainment analysis was performed by the programme coordinator. A web-based system was developed to assist the academic staff in measuring the PO and examining the programme performance. The system uses a mathematical model to calculate the PO scores automatically and efficiently. The process of developing the mathematical model is shown in Figure 2. Before the mathematical model was designed, the technique employed in the creation of the model was first discussed with the external examiner and statistician. Two techniques; the PO averaging and the De Morgan Probability theorem were examined by applying them on a few samples and the results were analysed. The optimum technique was employed in the model development. The model was then implemented in the programme assessment and its feasibility was examined.  It is important to select the suitable courses for the programme assessment before the PO measurement is carried out. The criteria used in the selection of courses were the core courses that directly addressed the programme outcomes and the minimum number of courses selected for each PO was three. In the PO measurement for an individual course, the PO score was measured from a few assessment components for example, the examination question, report and assignment that address the selected PO. In this sampling based approach, only 30% of the overall students were selected in the measurement if the total number of the students is 100 and above, and from this percentage, the proportion of good, moderate and weak students was 0.33. The PO for an individual course, PO I was calculated using Equation (1).
where N is the total number of assessments used in the measurement and P is the percentage of the average mark for each assessment, which is given by Equation (2).
where k is total the number of student involved in the measurement, M j is a mark scored by each student and M f is the full mark for the assessment. Table 2 shows example of PO score calculation for an individual course using the above equations. The overall PO score for multiple courses was computed using mathematical model which was derived from De Morgan Probability Theorem for mutually exclusive events as shown in (3). The De Morgan Probability Theorem has been used in researches, for example to proof the circuit lower bound for automatic learning algorithm and for detecting and diagnosing faults found in heavy-duty diesel engines [11,12]. In probability, events that are not associated together or occur separately are said to be mutually exclusive [13]. The rationale of using De Morgan Probability theorem is that the PO score for each course is measured from different students, activities, types of assessment and methods of assessment. Therefore, the PO score from multiple courses cannot be averaged. The probability of achieving the targeted level of programme outcomes, PO m is given by where m is PO's number, k is the number of courses selected in the measurement, P'(1) is percentage of course 1 not achieving the KPI, P' (2) is the percentage of course 2 is not achieving the KPI, P'(k) is the percentage of course k of not achieving the KPI.
Rewriting the equation using PO term and the probability theorem PO_C k = 1 -P ' (k) which is the percentage of achieving the KPI is equal to one minus the percentage of not achieving the KPI (Blitzstein et al. 2015), yields (4), a more detail elaboration of the overall PO calculation.
where PO_C 1 is the PO score for course 1, PO_C 2 is the PO score for course 2 and PO_C k is the PO score for course k. For example, referring to Table 3, if there are four courses were selected in the measurement of PO1 and the PO score for course 1; PO_C 1 is 80%, for course 2; PO_C 2 is 70%, for course 3; PO_C 3 is 60%, and for course 4; PO_C 4 is 50%, then using (4) the overall PO1 for the programme is 83.2%. The mathematical model was first implemented in 2012 where the performance of six programmes was evaluated based on the programme outcome. Before the mathematical model was used, it was tested on a few selected courses in a workshop by the Head of Programmes and Outcome Based Education (OBE) committee to ensure error free and correct calculation. Once it was confirmed that the mathematical model and the webbased system were functioning well, a workshop was conducted to train the course coordinators to use the web-based system for measuring the PO and for completing the CQI report which was generated automatically.
The PO attainment analysis was presented to the faculty management and discussed in the Closing de Loop (CDL) workshop every semester. Actions to be taken to improve the programme outcomes and to solve issues related to curriculum, teaching delivery and the implementation of OBE were identified in the pre-CDL workshop. These proposed actions were further discussed in the CDL workshop for approval. Monitoring of the actions taken was carried out by the management in the department and faculty levels.  Figure 3 shows one of the PO score obtained from the selected courses. Even though there is variation in the PO score, using the mathematical model, the overall PO can be determined. Figure 4 shows the PO attainment of one of the programmes which was obtained using the mathematical model and sampling-based approach. Each PO has exceeded the key performance indicator (KPI) target (a score greater than 65%) which shows the programme is successful.

RESULTS AND DISCUSSION
The percentage of student population in achieving the PO score of greater than 65% in one of the programmes is shown in Figure 5. More than 98% of the students have achieved the required knowledge and skills. These results demonstrate that the sampling based approach enables the achievement of the programme to be evaluated and the effectiveness of the OBE implementation in the programme to be examined.
The sampling based approach provides several benefits. With a small number of samples used in the work, the performance of the programme can be analysed and the percentage of the student population that meets the targeted KPI could be determined in the specified time frame. Sampling based approach has been used widely in several applications to provide clear picture on the performance and quality of the instruments and services. Blanco et al. [14] has used sampling based approach to identify quality of service for web service orchestration. Began et al [15] has employed sampling based approach to determine the planned starting time of each appointment.

CONCLUSION
The programme assessment process and the measurement of PO using the sampling based approach have been described in this paper. A mathematical model, which was derived based on the probability theorem, has been developed and used in the web-based system to assist in the final decision of the PO score. The implementation of the sampling based approach in the assessment provides a good indication on the programme and the students' performance. Using the criteria for the assessment, the sampling based approach demonstrates that the programme is successful and the students have gained the required knowledge and skills.