Framework for virtual education of COVID-19 vaccines for Mandarin-speaking learners: an educational intervention module

Background: In the United States, patients with limited English proficiency face significant barriers to comprehending and acting upon health-related information, particularly during the COVID-19 pandemic. The ability of health professionals to communicate COVID-19-related information to Mandarin-speaking patients has proved critical in discussions about vaccine efficacy, side effects, and post-vaccine protection. Methods: The authors created a one-hour educational module to help Mandarin-speaking medical students better convey COVID-19 vaccine information to Mandarin-only speakers. The module is composed of an educational guide, which introduced key terminology and addressed commonly asked questions, and pre- and post-surveys. The authors recruited 59 Mandarin-speaking medical students all of whom had previously completed a medical Mandarin elective. The module and surveys were distributed and completed in August 2021. Data analysis measured the change in aggregate mean for subjective five-point Likert-scale questions and change in percent accuracy for objective knowledge-based questions. Results: 86.4% of participants were primary English speakers with variable levels of Mandarin proficiency. The educational module significantly improved participants' subjective comfort level in discussing the COVID-19 vaccine in English and Mandarin. The largest improvement in both English and Mandarin was demonstrated in participants' ability to explain differences between the COVID-19 vaccines, with an aggregate mean improvement of 0.39 for English and 1.48 for Mandarin. Survey respondents also demonstrated increased percent accuracy in knowledge-based objective questions in Mandarin. Conclusions: This module provides Mandarin-learning medical students with skills to deliver reliable information to the general population and acts as a model for the continued development of educational modules for multilingual medical professionals.


Introduction
Limited English proficiency is known to be a major barrier to accessing quality healthcare within the United States 1 .Language barriers have contributed to a lack of access to accurate and updated healthcare information, a disparity that has been exacerbated by the COVID-19 pandemic 2 .Chinese-speaking communities comprise a large portion of the United States population, over 3.3 million U.S. residents, of which 55% meet the criteria for limited English proficiency 3 .This disparity in health literacy has created uncertainty and confusion around topics related to SARS-CoV-2/COVID-19, especially vaccination efficacy, side effects, and post-vaccine protection.Cooperation between medical professionals and community groups has proved essential throughout the pandemic, particularly in delivering critical information to individuals in their native or preferred language 4,5 .
In March 2021, Asian Center-Southeast Michigan, a non-profit organization serving the Asian-American/Pacific Islander community in metropolitan Detroit, contacted our group of Mandarin-speaking medical students at the University of Michigan Medical School to request a presentation in Mandarin about the COVID-19 vaccines for their Mandarin-speaking community members.While preparing the presentation, we realized that a niche set of medical terminology not covered in traditional medical language electives would be required to comprehensively communicate vaccine information.Additionally, few educational resources were available in Mandarin to assist learners to acquire the requisite vocabulary for conveying information about the COVID-19 vaccines.However, since all the presenters had previously completed the Medicine in Mandarin elective, we possessed adequate background skills to learn new medical terms necessary for effectively communicating information about the COVID-19 vaccines to Mandarin-only speakers 6 .We worked with the Medicine in Mandarin elective instructor at the University of Michigan as well as Mandarin-speaking staff at Asian Center-Southeast Michigan to identify, learn, and practice relevant terminology.We incorporated this set of vocabulary with previously learned content to accurately convey healthcare information during the presentation to community members in April 2021.
After the presentation, we created an educational module to share our group's experiences with other Mandarin-speaking medical students to improve communication of COVID-19 vaccine information to Mandarin-only speakers.Previous research has shown that Chinese language workshops and educational materials like phrasebooks allow medical students to refine their linguistic abilities and help to increase students' perceived confidence in communicating with Mandarin-speaking patients 7,8 .Additionally, peer-led and peer-developed curricula can readily identify knowledge gaps in an educational cohort while providing a collaborative space for language learning 7 .Our moduleincluded a table of key terms, responses to ten relevant "commonly asked questions," and pre-and post-surveys to evaluate the effect of our intervention.
This study seeks to evaluate the efficacy of an educational module designed to assist Mandarin-speaking medical students with acquiring and utilizing Mandarin terminology regarding COVID-19 vaccine-related information.Our educational guide acts as a model for developing topical phrasebooks or modules that can assist multilingual medical students with learning and practicing important terminology that they will use during conversations with patients.We believe that our instructional module addresses a gap in the medical language discipline by providing a guide for professional students to better tackle the language barrier that limits access to quality information amid the uncertainty of the COVID-19 pandemic.

Study design
The study was designed as an educational intervention via an online module, with participants' comfort level with discussing vaccine-related information being analyzed before and after engagement with the module.The location of research was centered in Ann Arbor, Michigan, though participants engaged remotely from all across the United States.The educational module was designed to be completely virtual.Surveys and corresponding data were collected for a two-week period in August 2021.Data were processed and analyzed by the research team in Ann Arbor, Michigan.

Study population
To normalize medical Mandarin knowledge and minimize selection bias, we limited our study population to students who had previously enrolled in and completed a medical Mandarin course at their respective medical schools.This requirement operated as a baseline for ensuring at least intermediate to advanced proficiency in Mandarin Chinese.Medical Mandarin electives are offered at medical schools across the nation and aim to teach accurate Mandarin terminology for use during patient encounters and introduce cultural values so that students provide culturally competent care for Mandarin-speaking patients.
Survey respondents were identified by consulting student rosters and email distribution networks for medical Mandarin electives at medical schools across the country.Our module was distributed via email to medical students at seven U.S. academic medical centers.Participants were identified by personal connections with Mandarin-speaking medical students at other academic medical centers.Participants were encouraged to distribute the module to other medical students who met the study's inclusion criteria.Inclusion criteria were limited to

Amendments from Version 2
In response to one of our reviewer comments, we critically reflected on the methodology of our qualitative results.We ultimately decided that our paper would be stronger and convey a more concise conclusion if we focused on the quantitative results.We removed the qualitative section: Table 3 and the corresponding texts in the manuscript.
Any further responses from the reviewers can be found at the end of the article students who had completed an elective in medical Mandarin at a medical school.A study size of at least thirty participants was determined to ensure adequate student representation and data significance.Participants were given two weeks to complete the module.The module was designed to be completed within one hour.The first fifty survey respondents were each compensated $40 for completion of the pre-survey, educational guide, and post-survey.Funding was provided through the University of Michigan Medical School M1 Summer Impact Accelerator Grant.

Module structure
The module comprised a pre-survey, educational guide, and a post-survey.All three elements were originally constructed in English.The educational guide contained vocabulary words and sentences that were then translated into Mandarin Chinese with accompanying pinyin for teaching purposes.The pre-and post-surveys were created and administered via Qualtrics.All survey questions were written in English.Each survey contained 18 questions.Six questions inquired about participants' comfort levels in addressing the stated learning objectives using English.Eight questions inquired about participants' comfort levels in addressing the stated learning objectives using Mandarin Chinese.Four objective questions tested participants' ability to select one correct answer about vocabulary to effectively convey COVID-19 vaccine-related information in Mandarin.On the post-survey, participants were asked six optional feedback questions.These questions comprised participants' subjective ratings of the efficacy of the module overall and educational guide, goals in learning Mandarin, and free-text boxes for assessing participants' comments about the module.Participants were expected to respond in English for all the survey questions.

Ethical statement
The study was exempt from regulation by the University of Michigan Institutional Review Board (HUM00200622), as it involved a benign intervention limited to responses by participants via online surveys.Participants' names and email addresses were collected for the distribution of compensation and were otherwise de-identified for all research purposes.All participants were required to sign a consent form agreeing to collection of information including name, email address, and personal demographics.Participants were informed that their information would be de-identified as part of data processing.All participants provided consent prior to enrollment by electronically signing at the beginning of both pre-and post-surveys.

Data collection
In engaging with our module, students were instructed to complete a survey before and after reading through a bilingual English-Mandarin educational guide that contained learning objectives, a table of key vaccine terms, and information regarding 10 commonly asked questions that we heard from the local community.A copy of the survey can be found under Extended data 9 .Our guide synthesized information from government sources, peer-reviewed research, and Michigan Medicine faculty.Our targeted educational module evaluated participants' knowledge and fluency of vaccine-related information in Mandarin via scores on the two surveys.The pre-guide and post-guide surveys were drafted on Qualtrics.The pre-survey contained questions about participants' demographics and language backgrounds.Both surveys included six questions regarding participants' comfort levels using English to discuss each of the learning objectives identified in the educational guide, and eight questions regarding participants' comfort levels using Mandarin to discuss each of the learning objectives.The questions were identical in pre-and post-surveys to measure a change in participants' comfort levels in using both English and Mandarin for each learning objective before and after intervention (Appendix B).Three optional free-text questions were provided in the post-survey for participants to leave feedback about the educational module.The module was reviewed by Mandarin-speaking medical faculty, a Mandarin healthcare interpreter, and two native Mandarin speakers from Asian Center-Southeast Michigan to ensure accuracy of the medical content and translation.

Analysis
We analyzed data for the Likert-scale questions using R Studio and created stacked bar plots for pre-and post-survey responses for each learning objective, categorized by English vs Mandarin.Responses were recoded into numerical values: strongly disagree = 0, somewhat disagree = 1, neither agree nor disagree = 2, somewhat agree = 3, strongly agree = 4 to allow measurement of change in mean.Scores before and after the intervention were compared using paired t-tests.The final analysis represented an aggregate change before and after intervention for each question regarding English and Mandarin comfort levels addressing each learning objective.
Data analysis for knowledge-based questions was completed using the GraphPad Prism 9 for visual representation and Chi-squared analysis.Bar plots were created for pre-and post-survey responses to four objective content multiple-choice questions.Two of the questions (6 and 7) allowed for multiple answer selection and included more than one correct answer.For these questions, each answer option was evaluated pre-and post-survey via Chi-squared analysis to assess for significant changes in answer selection after our intervention.For questions that had a single correct answer (Questions 5 and 8), only the participant selection of the correct answer was included in our data analysis.

Results
Fifty-nine students across the United States participated in the study, all of whom had completed a medical Mandarin elective.Participants represented varying classes in medical school, and over 86% of students reported English as their first language.Participants learned Mandarin from a variety of sources, with the most frequent being parents/family life, followed by weekend Mandarin classes, secondary or post-secondary education, and self-study.The full dataset can be found under Underlying data 9 .
Participants' goals for learning Mandarin included improving communication with Mandarin-speaking patients and family members, learning about personal heritage/culture, and pursuing an international career in a Mandarin-speaking country.Nearly half of the participants expressed interest in working with a community and/or patient population that is mainly Mandarin-speaking (Table 1).This educational module significantly improved participants' subjective comfort level in discussing the COVID-19 vaccine in both English and Mandarin.Participants' pre-and postsurvey subjective responses to various questions were recorded on a Likert-scale from "strongly agree" to "strongly disagree" (Figure 1).Comparing the pre-and post-survey responses, there was a statistically significant change in the mean response towards "strongly agree" on Questions 2 through 6 for discussing questions in English and Questions 2 through 8 for discussing questions in Mandarin (Figure 1).Specifically, after completing the module, students felt more comfortable using English to explain how COVID-19 vaccines protect against COVID-19 (Q2 English), the differences between available COVID-19 vaccines (Q3 English), side effects of the COVID-19 vaccines (Q4 English), the timeframe for COVID-19 vaccines to be effective (Q5 English), and what activities are recommended by the CDC after receiving the vaccine (Q6 English).Regarding understanding and explaining concepts in English, students demonstrated a 0.12 aggregate mean increase in comfort levels for Q2 (p<0.05),0.39 increase for Q3 (p<0.01),0.21 increase for Q4 (p<0.01),0.30 increase for Q5 (p<0.01), and 0.25 increase for Q6 (p<0.01).The largest difference in mean score was observed for participants' comfort levels with understanding the differences between the available COVID-19 vaccines in English (Q3), in which the aggregate mean of 3.20 increased to 3.59 post-intervention (Table 2).
After completing the module, students felt more comfortable using Mandarin to explain medical terminology (Q8 Mandarin), how COVID-19 vaccines protect against COVID-19 (Q9 Mandarin), the differences between available COVID-19 vaccines (Q10 Mandarin), side effects of COVID-19 vaccines (Q11  Regarding comprehension and explanation of vaccine-related concepts written in Mandarin, students demonstrated a 0.66 aggregative mean increase in comfort levels for Q8 (p<0.01),1.2 increase for Q9 (p<0.01),1.48 increase for Q10 (p<0.01),1.07 increase for Q11 (p<0.01),1.05 increase for Q12 (p<0.01), and 1 increase for Q13 (p<0.01), and 0.85 increase for Q14 (p<0.01).Similar to the largest improvement in mean score on the English portion of the intervention, participants displayed the most significant improvement in mean score for using Mandarin to explain the differences between available COVID-19 vaccines (Q10) (Table 2).
Compared to baseline, participants specifically reported an increased level of comfort for using both English and Mandarin to convey the vaccine-related information summated in our learning objectives.Overall, there was a significantly greater increase in mean scores for the Mandarin portion after participants engaged with the module with some mean score increases for the English portion as well.
There was no statistically significant difference between pre-and post-survey responses in English on Question 1, which measured students' comfort conversing with others using medical terminology in English.There was also no statistically significant difference between the pre-and post-survey responses in Mandarin on Question 7, which assessed students' comfort communicating with others in conversational Mandarin.These results were expected as the participants were English-speaking medical students who have enrolled in a medical Mandarin course during medical school.Given that this module focused on communicating COVID-19 vaccine information in Mandarin, it would not likely improve students' comfort using medical terminology in English or students' comfort in conversational Mandarin generally.
In addition to completing the Likert-scale questions, participants were asked four knowledge-based questions (Q15-Q18) in Mandarin under the section "Competency in communicating information regarding COVID-19 using Mandarin".Q15 and Q18 had a single correct answer, and participants demonstrated a statistically significant improvement post-intervention for both questions (p<0.01).Q15 asked participants to select the set of Mandarin terms that were most relevant for describing how the COVID-19 vaccine protects against the virus.After engaging with the module, the percentage of participants who chose the correct answer of "immunity, antibody, injection, vaccine" increased by 20.3%.An increase of 25.4% was also observed for participants who selected the correct answer to Q18, which asked participants to select the best answer to the question translated to "Can you still spread the virus after getting vaccinated?" Students could select multiple correct answers for Q16 and Q17 (Figure 2).Q16 asked participants to choose terms that  described the side effects of the COVID-19 vaccine.Participants demonstrated a statistically significant improvement in selecting two of the correct answers, "fatigue" (p<0.001) and "low-grade fever" (p<0.0001) as well as not selecting one of the incorrect answers, "death" (p<0.05).Q17 asked participants to identify specific activities that the CDC deemed appropriate after receiving the vaccine as of the time of the survey in April 2021.
There was a statistically significant increase in selecting one of the correct answers "Can travel internationally without quarantining after returning to America" (p<0.0001), and a significant improvement in not selecting one of the incorrect answers "Can attend medium or large gatherings" (p<0.05)(Figure 2).
Overall, 97% of participants recommended this educational module to others who want to improve their abilities to communicate with and educate native Mandarin speakers about the COVID-19 vaccine in Mandarin.

Discussion
Through this study, we demonstrate that topical educational modules can improve medical students' ease and comfort with communicating vaccine-related information to Mandarin-speaking patients.While many medical students were able to inform patients about COVID-19 vaccines in English, our module offered students appropriate instruction to convey that information to Mandarin speakers.Most participants reported significantly increased comfort with discussing vaccine-related information in Mandarin after completing the module.Additionally, participants reported greater comfort in explaining vaccine information in English, likely due to an increased fund of knowledge and vaccine updates provided by the Centers for Disease Control and Prevention and University of Michigan faculty.
The largest increase in comfort levels for both English and Mandarin was for the learning objective addressing differences between the US-based COVID-19 vaccines (Pfizer, Moderna, Johnson & Johnson), indicating a potential gap in knowledge that this intervention helped to mitigate.The efficacy of the educational module was also demonstrated with increased accuracy in selecting correct answers to knowledge-based questions in Mandarin, indicating that participants' knowledge base and ability to interpret health information in Mandarin had improved.
Our project was a student-developed and student-led initiative with the support of faculty and community partners for compiling medical information and ensuring accurate translation of the educational materials.Our team drew on personal experiences from the Medicine in Mandarin elective and basic science coursework at the University of Michigan Medical School to customize the educational module's content, length, and scope to maximize accessibility to medical students.Participants noted the module's point-by-point, self-paced structure, key terms table, and responses to 10 relevant "commonly asked questions" to be especially helpful.While many participants were already knowledgeable about the COVID-19 vaccines, we recognize that the true impact of our module centered around teaching students how to explain the information appropriately in Mandarin.
One limitation of our study is generalizability, as medical Mandarin electives may not be available to medical students across the nation, do not have a standardized curriculum, and cover a range of topics using various learning modalities.Many U.S. medical schools offer medical language electives so that their students can provide personalized care to patients.
Although we required participants to have previously completed a medical Mandarin elective, individual participants still possess varying degrees of proficiency that may have impacted comprehension of the module's content and subsequent performance on the survey.While our module improved participants' comfort level and objective ability to communicate COVID-19 vaccine-related information in Mandarin, it is outside the scope of this work to measure the health outcomes of patients who engaged with module participants.
We encourage the continued production of similar educational resources regarding COVID-19 as the pandemic continues to evolve, as well as other medical topics of interest.These modules could incorporate extensive pedagogical approaches such as schemas for patient education, audio recordings, or miniquizzes to check users' comprehension as they progress through the module.Another possible area for research is the formulation of a standardized medical language curriculum that provides baseline levels of linguistic proficiency as agreed upon by health professionals and language experts.In the age of digital education, we should strive to improve access to education and relevant materials for multilingual students enrolled at institutions without official medical language electives.Although professional medical interpreters remain necessary, the ability for providers to communicate with non-English-speaking patients in their native language is beneficial not only for building rapport but also when interpreters are unavailable, particularly in low-resource settings.Finally, it is important to assess the clinical utility that multilingual communication has on patient encounters to more effectively determine how information-sharing occurs during these medical encounters.Improving language proficiency in multilingual students and providers will help improve health literacy in patients with limited English proficiency by ensuring adequate communication of health-related information and providing culturally appropriate care in an inclusive environment.The latter is crucial as patient populations in the United States grow increasingly diverse.
In the context of widespread vaccine hesitancy among racial and ethnic minority groups in the United States, it may also be prudent to assess whether receiving information about the COVID-19 vaccine in one's preferred language increases the likelihood of taking the vaccine.Results of such a study might inform efforts to increase vaccination rates among patients with limited English proficiency.
Ultimately, this Mandarin educational module significantly improved participants' subjective comfort level in discussing the COVID-19 vaccine with Mandarin-speaking patients.Our module provides Mandarin-learning medical students with the skills to convey crucial health-related content to patients and functions as a model for developing additional educational resources for multilingual students.For non-US readers, it is not clear (to me at least) -do students typically take languages as electives in Medical School?Or is this unusual?

○
In the Discussion you say that this was a student led and student driven initiative, but that is not apparent from the Introduction, where it seems to be a Faculty designed and implemented intervention?

○
Could the materials you have prepared for the module be made publicly available, so that others can use them?It could be a great way to have an international collaboration going, if readers contacted you and you could pool data and design studies where you test the patients' experiences as well as the students'?

○
In the Methods -"data was" = "data were" (data is plural).

○
The issue of bias, in the Methods -does it really matter?You are not claiming representativeness, and could not possibly do so, so to me, the matter of bias is not important and could be deleted.

○
For the analysis -How were the open-ended questions analysed?I think qualitative content analysis or content analysis would have been appropriate -but it seems you might have "cherry picked" quotes from the data, which is not really considered best practice.You might want to re-analyse the written responses?

○
In the Discussion you say that this was a student led and student driven initiative, but that is not apparent from the Introduction, where it seems to be a Faculty designed and implemented intervention?RESPONSE: In the Discussion section, we have added context about the faculty and community partners that assisted with compilation of health information and translation of educational materials.

○
Could the materials you have prepared for the module be made publicly available, so that others can use them?It could be a great way to have an international collaboration going, if readers contacted you and you could pool data and design studies where you test the patients' experiences as well as the students'?RESPONSE: The module and data sets are provided in the Dryad link located in the "Data availability" section toward the end of the paper.

○
In the Methods -"data was" = "data were" (data is plural).RESPONSE: Corrected in the text

○
The issue of bias, in the Methods -does it really matter?You are not claiming representativeness, and could not possibly do so, so to me, the matter of bias is not important and could be deleted.RESPONSE: Corrected in the text ○ For the analysis -How were the open-ended questions analysed?I think qualitative content analysis or content analysis would have been appropriate -but it seems you might have "cherry picked" quotes from the data, which is not really considered best practice.You might want to re-analyse the written responses?RESPONSE: We address that the open-ended questions were analyzed for prevailing themes that characterized the 107 unique responses.

○
You indicate in the Results that 59 students participated -I am guessing there is no way to know how many Mandarin speaking/studying students there were at the time of the study, to give the reader a feel for response rate/participation rates?RESPONSE: All students had completed a medical Mandarin elective and had some degree of Mandarin proficiency prior to completing the study.

○
Apart from the clarification needed, overall an interesting piece of work -and as noted, if you could share the module for the intervention, that would be really helpful, I am sure, to others, and could result in collaborations for you and them.RESPONSE: The module and data sets are provided in the Dryad link located in the "Data availability" section toward the end of the paper.

Figure 1 .
Figure 1.Response distribution to pre-and post-survey questions addressing participants' comfort levels for discussing learning objectives in English and Mandarin from our virtual educational module for Mandarin-speaking medical students.

Figure 2 .
Figure 2. Response percentages for pre-and post-survey objective questions from our virtual educational module for Mandarinspeaking medical students.For Q16 and Q17, correct answers are graphically represented as answers A, B, and C, while incorrect answers were recorded as X, Y, and Z.
under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Table 2 . Change in aggregate mean for each learning objective in English and Mandarin. The table
demonstrates the change in aggregate mean for each learning objective in both English and Mandarin.Participant responses were recoded into numerical values 0-4 (strongly disagree-strongly agree) to calculate aggregate mean score, with a shift toward more positive values, indicating greater comfort level with learning objectives.Statistical significance calculated using paired t-test.* indicates P-value <0.05, ** <0.01, *** <0.001, **** any value beyond <0.0001.NS denotes not significant.

the rationale for developing the new method (or application) clearly explained? Partly Is the description of the method technically sound? Partly Are sufficient details provided to allow replication of the method development and its use by others? Partly If any results are presented, are all the source data underlying the results available to ensure full reproducibility? Partly Are the conclusions about the method and its performance adequately supported by the findings presented in the article? Partly Competing Interests:
You indicate in the Results that 59 students participated -I am guessing there is no way to know how many Mandarin speaking/studying students there were at the time of the study, to give the reader a feel for response rate/participation rates?Apart from the clarification needed, overall an interesting piece of work -and as noted, if you could share the module for the intervention, that would be really helpful, I am sure, to others, and could result in collaborations for you and them.No competing interests were disclosed.

confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Corrected in the text Conclusions section of the Abstract -and earlier in the Methods -and throughout the paper -I found it confusing, to understand whether these were native speakers of Mandarin, or otherwise, and it's not until much later in the paper it becomes clear.It would be good to make this clearer, earlier.RESPONSE: We have clarified this point in our abstract by providing context about the percentage of participants that are primary English speakers For non-US readers, it is not clear (to me at least) -do students typically take languages as electives in Medical School?Or is this unusual?RESPONSE: In the Discussion section, we have added context about medical school language electives.
RESPONSE: We have reflected this comment in the introduction.Thank you for your recommendation to broaden the scope of struggles with English proficiency.○ Results section of Abstract -"demonstrated in the participants ability" should read ○ "demonstrated in the participants' ability" RESPONSE: ○