Assessing Midwifery and Medical Student’s Interprofessional Learning: the Use of Portfolio

Received Jan 26, 2018 Revised Apr 4, 2018 Accepted Aug 27, 2018 To evaluate the portfolio in assessing interprofessional learning between medical and midwifery students. The student portfolio is assessed using a rubric consisting of four assessment criteria. A total of 32 student portfolios were tested for reliability coefficients and interrater agreements. We conducted an in-depth interview with mentors and focus group discussion (FGD) with students for exploring their perceptions of the ability of the portfolio to assess the learning. Interview and FGD data were converted into verbatim transcripts then were analyzed by two coders using open coding techniques. The reliability coefficient is 0.808. Inter-rater agreements for each assessment criteria are ranging from moderate to high. Mentors and students have positive insights toward the assessment system. This study supports the use of portfolios as an interprofessional educational assessment tool.


INTRODUCTION
The implementation of interprofessional education (IPE) in the health profession education is expanding globally. This is supported by the agreement between various health professions to develop consensus on interprofessional core competencies as the basis for curriculum development and interprofessional learning. [1] However, assessing interprofessional competences is quite challenging. [2] It needs robust assessment system including standardized tools, longitudinal assessment, and variety of data. [2][3][4] It also demands a system that support students to develop the knowledge, skills, attitudes and values to work colaboratively in an interprofessional team [5][6].
A portfolio is an assessment method that can describe the authenticity of learning. It also can encourage the student to be responsible for the learning and development of his or her own competence by collecting evidence of learning experience or performance [7][8]. This method has widely been used in various contexts of medical education, including undergraduate, professional, postgraduate and sustainable professional development students [8][9]. Buckley et al. [10] state that the portfolio has the effect on learning, i.e., able to improve the ability of the integration of theory and practice, reflection and introspection, and selfstudy. The use of a portfolio in community-based medical education can stimulate reflective learning and underpin the program planning and supervision process [11]. The study of portfolio use in interprofessional learning is still limited [3]. Domac et al. [3] concludes that student reflection on the portfolio can illustrate the achievement of student interprofessional competencies.
In Indonesia, as one of the institutions that wish to prepare health workers ready for collaborative practice, the Faculty of Medicine of Universitas Sebelas Maret (FM-UNS) has developed a community-based IPE learning module for undergraduate medical and D3 midwifery students. This module is a pilot project to incorporate IPE into medical education curricula (IPE FM-UNS team). It uses family health services, especially maternal and child health as the context of its learning with a four week learning time. The learning objectives have been developed based on the four core competency domains by the Interprofessional Collaborative Practice (IPCP) team as a guide for developing the IPE curriculum [1]. For assessing the students' IPE learning, we design a portfolio with the rubric as the IPE portfolio assessment guide.
To assess the quality of the IPE portfolio, evaluating of the scoring system with specific criteria is required [12][13]. So that the assessment system and the learning program may be accountable to the public [14]. Validity and reliability are among the evaluation criteria that aim to determine whether the assessment system indeed assesses the content or competence to be tested or not [15][16].
Interestingly, there are no significant researches that studied the use of portfolios in interprofessional learning of midwifery students. So far, the study of portfolio evaluation has been limited to only one health profession (medical or midwifery only) [11,[17][18][19][20][21], while portfolio evaluation on interprofessional learning is still limited [3].
This study evaluated the IPE portfolio content by exploring the views of faculty lecturers as IPE portfolio assessors and investigating student perceptions as the subjects of IPE learning. It also reviewed the reliability of the IPE portfolio rubric.

RESEARCH METHOD
A mixed methods research design was done in this study, combining quantitative and qualitative design with the aim of gaining a broad understanding of research questions. [22] To evaluate the reliability if the IPE portfolio, cross-sectional survey strategy was chosen, while case study strategy was used to explore the views of supervisors and students. In case study strategy, a program, event, or activity was in-depth explored by using a variety of qualitative data collection procedure [22].

Sample and Setting
For the quantitative data, the sampling method was total sampling. All results of the portfolio rubric were included in this study, due to the small population (n = 32, undergraduate medical students = 17, D3 Midwifery students = 15). For the qualitative data, there were 17 IPE students (n = 17, medical students = 10, D3 midwifery students = 7) and all mentors/assessors (n = 4) who were the samples in this study.

Portfolio Assessment Procedures
A portfolio was used to assess the interprofessional learning. Undergraduate medical and D3 midwifery students attended in the learning with the context of maternal and child health services. The portfolio consisted of four sections based on the learning period in week 1 to 3 and ended with the final reflection in week 4. During the learning, each group consists of 5-6 students were guided by two mentors, each of whom was a lecturer of medicine and midwifery. Each week students would do mentoring that aims to stimulate reflection, monitor learning, and provide feedback. At the time of assessment, each student portfolio will be assessed separately by two assessors who constitute one midwifery and one medical mentor from another group using the criteria-based assessment rubric. Both assessors had been trained to use the assessment rubric before assessing the students' real IPE portfolio. The students' portfolio was assessed using the following four criteria of assessment: 1. Level of reflection 2. Use of Evidence of Learning 3. Use of relevant References 4. Quality of work plan Based on the four criteria above, the students' portfolio was assessed objectively using a quantitative scale of 0 to 3, and the total value was 12. Portfolio assessment was done in each section so that four subvalues were added to the final value. At the beginning of interprofessional learning, the students have received an explanation on how they were assessed using the portfolio assessment rubric.

Data Collection and Analysis
The data collection procedure in this research consisted of three forms, one form of quantitative design and two forms of qualitative design. In quantitative design, portfolio assessment results were employed to calculate the validity and reliability of the rubric. Data analysis used the SPSS software for windows edition 21.0 or the data analysis. In qualitative design, an in-depth interview was done to faculty mentors and Focus Group Discussion (FGD) students as a method of data collection. This design is under the purpose of case study strategy, which is for in-depth exploration of a program (22) which, in this study, it was the use of portfolio as an instrument of assessment of interprofessional competence. The recording of in-depth interview and FGD was then converted into verbatim transcripts followed by data analysis with content analysis techniques using Atlas.ti for windows software. After the data preparation and selection of unit analysis, the data were read, interpreted as a whole and coded. The coding process was done by two medical education experts using open coding technique. Both coders examined one transcript of the same in-depth interview mentor and one transcript of the same FGD student separately, and subsequently, the level of agreement between the code of analysis was calculated until it reached an adequate level of agreement (Kappa = 0.87). Finally, the coders analyzed the remaining transcript consecutively until forming code, group, category and meaning [23].

Ethical Consideration
The ethical clearance to use student IPE portfolio data issued by the Ethics Committee of RSUD dr. Moewardi Surakarta, Central Java, Indonesia. Students and lecturers had been informed that participation in this study was voluntary. Also, the students were well informed that the participation would not affect their study grades.

RESULTS AND ANALYSIS 3.1. Characteristics of respondents
There were 32 total portfolio scores, each of which consisted of 4 sub-scores and had calculated the reliability and validity of the portfolio rubric. Characteristics of Respondents shown in Table 1.

Rubric Reliability
The data analysis was performed using SPSS for windows edition 20.0 against 32 total portfolio scores. The reliability coefficient of the portfolio section showed a high score (α = 0.808) (shown in table 2).  Table 3 shows the quality of each item of assessment criteria calculated by an interrater agreement analysis. It was found that all criterion items were relevant to an acceptable interrater agreement, ranging from 0.331 (moderate agreement) to 0.731 (excellent agreement).  [24][25]. The interrater agreement can be received with a range of moderate to high. It represents the extent to which the data collected in this study are correct representations of the variables measured [25].
Reliability is one of the attempts to standardize the portfolio, but the rigorous standardization can reduce the portfolio's own objectives, i.e., learning autonomy [9], [26], [20]. However, a low level of reliability is also a threat to the validity of the judgment itself [27]. Therefore, there is a need for balancing to achieve portfolio objectives, but portfolio reliability remains cultivated to the acceptable level [11], [21]. Inter-rater reliability is one of the critical issues that determine portfolio quality [11], [20], [28]. There are several strategies for achieving adequate inter-rater reliability, including the use of standardized assessment criteria, the use of assessment rubrics, the presence of a trained assessment committee, the number of assessors more than one, and an open discussion among assessors [29][30]. In this study, a portfolio assessment framework consists of 4 assessment criteria, training of all members of the portfolio assessment committee along with simulations, a two-person assessment committee to assess one student, a discussion, and negotiation between assessors before and during the process of determining the portfolio scores guided by the portfolio assessment rubric.

Mentor/Assessors' Perception
All mentors and portfolio assessors (n = 4) followed in-depth interviews, consisting of 2 lecturers of Medicine (Md) and two lecturers of Midwifery (Mw). There were several themes related to the validity of portfolio content and other themes regarding the use of portfolios to assess interprofessional competencies. The themes are presented with supportive quotes. All mentors and assessors agreed that a portfolio is an assessment method capable of assessing interprofessional competencies:

"I think it is able to assess interprofessional competence." (Mw1) "Yes, I think it is appropriate to assess interprofessional competence." (Md1)
In this case, the mentor also admitted that the student portfolio illustrated the development of interprofessional competencies during the learning process: "Yes, it can describe how the division of roles and responsibilities is conducted. For example, in the first week, a student said that she did not understand at all with this activity ..., then she tried to learn and to know each other ......., then she was able to divide her task and to fill each other's role while serving the patient". (Mw1) "Yes, it reflects an increase in interprofessional competence. Thus, they write in the portfolio how initially in the first week they did not know the other students, and then as long as they communicated and when they went together to the location, they did things together, shared duties when meeting patients and their families, and wrote that they could already reflect how to collaborate there."(Mw2) "The sections 1 to 3

have given rise to interprofessional competencies. If we see them writing from the process of writing reflection section 1, section 2, section 3, they have shown improvement and (the reflection) is getting deeper". (Md2)
Majority of assessors did not find it difficult to make an assessment based on an existing portfolio rubric: In a portfolio assessment, all mentors and assessors agreed that the mentor role was critical to guide students in collaborative learning. In addition, a scheduled mentoring system was also required so that the student portfolio could be as expected: "If for the mentoring process, yes it is essential...... We should guide the students per week to remind them to write a reflection in their portfolio....also we can ask whether writing reflection is difficult and we can also help them". (Mw1) Mentors also agreed that an individual mentoring system was required to provide specific feedback: Mentors and assessors agreed that portfolio writing could stimulate student ability in library search: "She was able to search for more references, study the journal according to what problems she found." (Md1) In addition, the mentors considered that one of the difficulties of writing portfolio was the difference in the depth of the curriculum of each profession so that the quality of student reflection was different: "…..if students in medicine, they are familiar with the scientific culture that we teach to find articles and write scientific papers, then it is beneficial in filling the portfolio. In contrast, if midwife students, I think they are not used to (reflect), so the reflection is also different (the quality)". (Md2) To improve portfolio quality, the mentors and assessors perceived that direct performance-based assessments could be included as evidence of learning: In-depth interviews with mentors and assessors illustrate their views on the use of portfolios to assess interprofessional learning. It is an aspect of face validity and content of the interprofessional learning portfolio. All mentors and assessors feel that the student portfolio illustrates the collaboration and improvement of interprofessional competencies. In this study, mentors are the primary source of information related to the development or achievement of student learning because they longitudinally follow, guide, assess and provide feedback from the beginning to the end of learning. [20], [31] One of the lessons for faculty is the on-going need for timely feedback and support to understand how to reflect, especially in the mentoring session. This study also shows that a more intense scientific learning experience in the educational curriculum also affects the quality of student reflection on the portfolio.

Student's Perception
A total of 17 students attended the focus group interview (n = 17, medical students (D), n = 10, midwifery students (W), n = 7) divided into 2 groups (G1, G2). There were two main themes related to face validity of the portfolio and the use of an IPE portfolio. Themes are presented with supportive quotes. The majority of students agreed that portfolios could be used to assess interprofessional collaborative learning: "yes it could be if for IPE and collaboration (used to assess interprofessional competencies)" . (W1, G1) In addition, the students also mentioned some of the interprofessional competencies that could be described in their portfolios, among others, the competencies of interprofessional communication, interprofessional ethics, roles and responsibilities, and teamwork: "In our portfolio, we also write about this collaboration, about the skills and attitudes we learn, which we will ultimately know that the way we behave and appreciate other professions is fundamental. In the portfolio, it is also written about how we can foster good communication between doctors and midwives. Also, the division of authority exists (in the portfolio)…….". (D7,G2) By reflecting on each part of the portfolio, the students felt compelled to do a better interprofessional collaboration. The students also felt that writing reflection on the portfolio helped them review the problems systematically: "From writing that portfolio, it makes me easier to review the problem in order. So, for example, this issue should be filled like this in the portfolio based on the questions. So, we will write so that one day we will not forget to plan, and how to solve the problem". (D3,G1) However, the students also recognized that writing portfolios were a tricky thing for them because it was the first time they did it: "because this is the first time, writing the portfolio is a scourge for us. It seems that writing reflecting on ourselves or writing down what we have done is easy, but when we want to write, it is usually difficult ". (W7, G2) The students also recognize that the main obstacle in writing the reflection and working on the portfolio was because the portfolio was individual and there was confusion about whether, to be honest, or to write something ideal.
"…. It is more to the discussion of the portfolio because I think the portfolio is a personal thing, everyone has their own problems, have different perspectives ". (D8, G2) "…..we are confused to make this portfolio. We write the portfolio for the real score, or we are told to assess ourself… honestly". (W3, G1) In the process of portfolio work, the students knew how they were assessed and worked according to the portfolio assessment section: As an additional part of the portfolio assessment, the students felt that they also needed performance assessments directly by trained instructors : "…For me, I like the portfolio to be an IPE assessment, but it should be accompanied by assessment like an observation". (W7, G2) "…..assessment is not only from portfolio but also from direct observation by the supervisor." (D1, G1) How students interpret events on interprofessional learning as a trigger of reflection is one essential factor in determining the acceptance of the portfolio as an IPE assessment method [3]. The ability to reflect, assess the shortcomings and strengths of self, and analyze the roles and responsibilities of the team is an important factor in the health care team [7], [9], [10]. Students recognize that their portfolio describes how they learn to achieve interprofessional learning goals. This portfolio also encourages them to collaborate with other students from different health study backgrounds better. On the portfolio, students will collect authentic learning evidence that illustrates the attainment of their learning objectives. Therefore, it is necessary to explore students' perceptions about whether the portfolio assesses the achievement of learning objectives or not [20], [32][33].
This study supports Domac et al. [3] which states that the portfolio can be used as a method of assessment of interprofessional learning. In addition, the portfolio also has a learning effect in which it can encourage students to reflect and collaborate with other professional students.

CONCLUSION
Based on this study's result analysis, a portfolio is a valid and reliable tool for assessing interprofessional learning regarding its ability to describe interprofessional learning goals and achievement when seen by the students and to assess interprofessional learning and competencies when seen by the assessors. However, the number of samples in this study is relatively small, so further research is needed by using larger sample quantities. Although there has been evidence through qualitative study of students' and mentors' perceptions of the ability of the portfolio to assess interprofessional competencies, further analysis of the contents of the portfolio itself, including student reflection and the evidence of learning needs to be done to prove whether the correct writing of reflection and the evidence really describes the achievement of interprofessional competence.