Evaluation of Game-Based Learning in Cybersecurity Education for High School Students

ABSTRACT


INTRODUCTION
National infrastructure, corporations, and agencies have fallen victims to cyber-attacks, and millions of sensitive data records have been compromised, especially in financial and healthcare sectors [1], [2]. Such security breaches not only result in substantial financial losses, but also greatly hurt the confidence of customers, business partners and stakeholders [3]. To protect against the cyber threats, a significant demand for skilled cybersecurity workforce is predicted in government and industrial sectors. Cybersecurity is a shared mission between government and industry, because a large portion of the national cybersecurity infrastructure is in the private sectors. Cybersecurity workforce development is the key to assuring that a nation has adequate security measures to protect and defend information and information systems. However, a global shortage of "1.8 million cybersecurity professionals by the year 2022" has been estimated [4]. According to the U.S. Bureau of Labor Statistics, the growth rate of jobs in information security is projected to be 37% from 2012-2022, and at the same time more than 209,000 cybersecurity jobs in the U.S. are unfilled every year. The increasing demand for cybersecurity professionals from both government and private sectors makes it a critical mission for higher education institutions to attract and train next generation of cybersecurity workforce and citizenry who are capable of advancing national economic prosperity and security.
Research indicates that students receiving computer education in high school are 8 times more likely to major in a computer degree. Nevertheless, the introductory secondary school computer science courses have decreased in number by 17 percent from 2005 and the number of Advanced Placement (AP) Computer Science Courses has similarly decreased by 33 percent [5]. A consensus report from The National Academies of Sciences, Engineering and Medicine suggested that "to prepare students for expanding role of computing in academia, and industry, government agencies and states should support local, state and national programs for computing education for the purpose of increasing exposure to computing, computational principles, information security and data analytics thoughout the K-12 pipeline" [6]. These statistics and reports indicate that the key to produce more graduates in the cybersecurity field is establishing a meaningful pathway earlier in the educational process. A primary challenge to achieving this goal is the lack of age-appropriate cybersecurity curricula implemented with pedagogical methods that are most conducive to learning at the high school level [5]. The continued increase in undergraduate and graduate enrollment in computing programs is consistent with the interests of governments in nurturing Science, Technology, Engineering and Mathematics (STEM) disciplines. Nevertheless, the increased enrollment of international students has contributed most in computing related graduate programs. It will be helpful to increase the interest of domestic (US) students in pursing graduate degrees in computing programs. [7] Studies have shown that students learn only 20% of what they hear and read, but can learn 90% of what they have practiced [8]. A study conducted by the Kansas State University showed that adults learn best when they are active partners in the learning process. The author of the study urged educators not to lecture adult learners but rather involve learners in discussions, problem solving, and hands-on activities [9]. As Anzalone, Poudel, and Vincent also indicated, "Hands-on activities and challenge tests enhanced students' interest, motivation, and ability to think critically about contemporary environmental issues in the region" [10]. Learners at the high school level experience greater cognition if they are given opportunities to actively engage in classroom activities that support the development of critical thinking and problem solving skills. Therefore, there is a critical need for an innovative curriculum and pedagogical methods in the area of cybersecurity education. One of the most emergent and rapidly growing filed in computer-based education is "game based learning." Game based learning include virtual reality games, web-based games, multi-user virtual environments, massively multiplayer online games, and simulations [11]. In game based learning, players can explore relevant aspect of games in a learning context designed by the instructor. In an effective game-based learning environment, a student can choose actions and experience the consequences of those actions along with the game play. Learners can make mistakes in a risk-free setting, and through experimentation, they actively learn and practice the right way to do things. However, till date, applying game based learning instructional method to cybersecurity education is limited [12].
U.S. Congress passed major legislative proposals to enhance U.S. cybersecurity and combat cyber threats. The U.S. Congress has urged that it is critical to develop high-quality educators to expand cyber education at early age [13]. To increase K-12 students' interest and and raise their awareness in cybersecurity, the National Security Agency (NSA) and the National Science Foundation (NSF) have jointly funded more than 300 summer camps to K-12 students and teachers across the nation for the past 3 years [14]. Purdue University Northwest (PNW), a NSA/DHS designated National Center of Academic Excellence in Cyber Defense Education, has successfully launched four GenCyber summer camps for 181 high school students with 51.3% of underrepresented minority (Africa Americans and Hispanics) ratio. PNW GenCyber camp developed an innovative game based cybersecurity education modules to provide high school students with hands-on activities in an immersive learning environment. PNW summer camp activities were delivered in the format of game based learning and hands-on labs. Four cybersecurity education games were developed to teach social engineering, cyber-attack and defense methods, secure online behavior, and cybersecurity principles. The game based cybersecurity education is extremely beneficial to the future cybersecurity workforce by exposing more high school students to the cybersecurity education pathway at a time when they are making decisions regarding higher education. Survey result of 154 camp participants indicated that the cybersecurity education games were very effective in cybersecurity awareness training. The innovative pedagogical methods and age appropriate game based learning curriculum, has made cybersecurity concepts more accessible to students of varying ability levels.

RESEARCH METHOD
The primary goals of PNW GenCyber high school summer camp are: 1) increase interest in cybersecurity, 2) raise general awareness of cybersecurity and help all students understand appropriate and safe online behavior, and 3) increase diversity in the US cybersecurity workforce. To achieve aforementioned Gencyber goals, we developed four cyber security education games for high shool students. The rationale of game topics selection will be described in the following sections. Futhermore, to evaluate the effectiveness of the developed cybersecurity education games, post-camp survey was conducted after the completion of summer camp activities. The survey questions and study protocol were carefully reviewed by the Purdue Internal Research Board to protect minor high school students. The game topics were selected based on the following reasons: a. Game topics were selected based on the priority and importance announced by NSA GenCyber program.
The request for proposal (RFP) from NSA GenCyber clearly indicated that appropriate and safe online behavior and 10 cybersecurity first principles are the top priority of GenCyber program. Threfore, we developed 3D secure online behavior game, and 2D GenCyber first principle card game. b. One game topic was selected based on the insight and experience of principle investigators. Quite a lot of cyber attacks start from social engineering. Social engineering is a way to manipulate and deceive people to acquire confidential information for further cyber attacks. Threfore, we developed a 3D social engineering game to prepare high school students to overcome social engineering scams. c. Another game topic was selected based on the game player's interest. Tower defense game is a sub-genre of Realtime Strategy (RTS) game, which is one of the most played game genre for high school kids. We developed cyber security tower defense game to allow the high school students learn basic cyber security knowledge by playing realtime cybersecurity tower defense game.
In the following section, we will describe each cybersecurity game in detail: 1. Social engineering game: Social engineering is the art of manipulating people so they give up confidential information, and social engineering scams such as phishing email have been extremely effective in security attacks. PNW GenCyber camp implemented a 3D VR game to simulate Piggybacking, Tailgating, and Mantrap in a security enhanced office environment to raise general awareness of social engineering scams. 2. Secure online behavior game: Secure online behaviors include: identifying phishing emails and appropriately handling them, distinguish between trustworthy web links and insecure links, handling phony phone calls, and protecting personal information. A 3D VR secure online behavior game was developed to simulated high school computer lab and student's bedroom environment. The secure online behavior game allows students to appropriately handle email messages, text messages, web links and phone calls, using various computing devices such as school computers, mobile phones, laptop computer, and networked game console. 3. Cyber Defense Tower Game: Tower defense game is a subgenre of strategy game to defend a player's territories or possessions by placing defensive structures on or along their path of attack [15]. A Cyber Defense Tower Game was created to allow students to protect their virtual computer server from the different cyber-attacks by applying GenCyber first principles and cybersecurity knowledge. Students need to select the correct types of defense tower to stop each wave of cyber-attacks. As the game progresses, the combinations of the different cyber-attacks would come faster and will make it more difficult for students to defend their servers. 4. 2D GenCyber Card Game: The Gencyber card game is a computerized version of physical GenCyber card game. Physical GenCyber card game requires two players to play the game in face-to-face mode.
The computer based GenCyber card game is a single-player version of the GenCyber card game, which allows the student to play the card game by themselves at any convenient time. The 3D characters of the game were created from Adobe Fuse software. Instead of modeling a 3D character from scratch, Adobe Fuse allows a user to assemble a 3D character from more than 20 base characters and further customize it into a unique character with different weight, height, skin tones, and texture. The 3D character from Adobe Fuse was transferred seamlessly into Mixamo software. Mixamo will automatically rig the 3D character and provide hundreds of different motion clips that can be used to animate the character. We selected essential motion clips (such as idle, walk, run, talk, sit, and stand) for each character and export the animated 3D character to Unity3D game engine.
The game environment was modeled mostly using the loyalty-free 3D assets from Unity Marketplace. Several 3D assets that related with social engineering and secure online behavior were modeling using Autodesk 3D Max and Maya software. The behaviors of the 3D game characters were implemented by programming Unity C# script for each 3D character and dynamic assets in the game environment.

Development of Cyber Tower Defense Game in Unity3D
The cyber tower defense game was implemented on top of Tower Defense Toolkit (TDTK) developed by Song Tan [16]. TDTK is a C# coding framework for the easy construction of Tower Defense games. TDTK comes with a bundle of scripts that can be adjusted to fit a variety of Tower Defense gameplay scenarios. The toolkit is customizable, and user can integrate their own art assets to make unique Tower Defense game. We have created 7 unique cyber-attacks ( Figure.   One limitation of TDTK is that there are only two types of attacks: ground, air and three types of defense towers: ground, air and hybrid. It's impossible to use the built-in attacks and defense towers to simulate the cyber-attacks and defenses. We have customized the Tower Defense Toolkit to include 7 additional attack types (Virus, Phishing, Trojan, Spyware, Ransomware, DDoS, and Sniffer), and 6 additional defense towers (Antivirus, Password, System Update, Secure Cyber Behavior, Encryption, and Firewall). Some defense towers will defend single type of attack, while other defense towers can defend multiple attack types. For example, the Antivirus tower will defend against Virus, Trojan and Spyware. In addition, certain type of attack can be defended by multiple defense towers. The Sniffer attack can be defended by Password tower and Encryption tower. Cyber Defense Tower Game contains three difficulty levels: tutorial, intermediate and competition level.

Single-player GenCyber Card Game
The single player version of GenCyber Card game was developed to enhance the students' understanding of 10 Cyber Security First Principles. The original card game was designed and created by Dr. Vincent Nestler at California State University San Bernardino [17]. The computerized GenCyber card game was created by scanning the cards and uploading the images into Processing programming environment. The site team observed students playing this game during the down time at camp and some students were even creating "cheat sheets" so they could beat their friends' times.

Post-camp Survey Design
An anonymous post camp survey on cybersecurity knowledge and career interests has been conducted to evaluate the effectiveness of the summer camp activities. The survey was conducted independent of the camp activities and the participation of the survey is totally voluntarily. The survey questions were designed to collect the following data: 1. Demographic data of the participants, activity participation rate, lab/game results; 2. Students' assessment of self-efficacy related to cybersecurity knowledge and the GenCyber camp experiences; 3. Consent form signed by the participants or guardians, and assent form from the Minor students.  I enjoyed learning about cybersecurity Q4.
I would like to learn more about cybersecurity Q5.
The teachers/faculty in this program made me more interested in cybersecurity Q6.
I know what cybersecurity means Q7.
I know more about cybersecurity than I did before this camp Q8.
I know more about computer science than I did before this camp Q9.
I am more comfortable learning cybersecurity concepts now Q10.
I know more about information security than I did before this camp Q11.
I can explain why cybersecurity is important Q12.
Overall this camp was a good experience Q13.
I am glad I attended this camp Q14.
I would like to attend more camps like this Q15.
My opinions and ideas were respected in this camp Q16.
I found the camp activities interesting Q17.
I liked interacting with the teachers at this camp

RESULTS AND ANALYSIS
Purdue University Northwest has successfully launched two 1-week summer camps in June 2016, and another two 1-week summer camps in June 2017. A total of 181 high school students attended the with 93 students (51.3%) were underrepresented African American and Hispanic students. During the 1-week summer camp, each student participated in activities mentored by a team of PNW faculty members and student assistants. The students were exposed to cybersecurity first principles and cybersecurity awareness by playing various cyber security computer games during the first two days. About 25 students in one classroom have competed for 3D Secure Online Behavior game, Cyber Defense Tower game and Single-player Gencyber Card game. We picked one winner from each competition and awarded the winner with a small gift on the last day of the summer camp. The post-camp survey was conducted at the last day of the summer camp. A 5-point Likert scale was used to measure the student's satisfactory rate of camp activities and experiences ranging from 5 (strongly agree) to 1 (strongly disagree). The post-camp survey of 154 camp participants (Table. 3) indicated that game based learning for cybersecurity enhanced student's knowledge in cybersecurity, and understanding of the cybersecurity first principles, and educated a digital citizenry with security awareness, and motivated them to pursue higher education and careers in the field of cybersecurity. The authors further analyzed the survey data to investigate the gender and racial difference of participants' experience in game-based learning. Hypothesis testing for two population means with unequal variance (T-Test) was used to test gender and racial difference for each survey question. For each survey question, we performed both two-tail and one-tail hypothesis test. For example, the hypothesis for "Qi. I enjoyed learning about computer science" was: 1. Two-tail hypothesis test H 0 : There is no difference in mean survey rating of Qi between female and male students (μ1 = μ2). H A : There is difference in mean survey rating of Qi between female and male students (μ1 ≠ μ2). Significance level (α)= 0.05 2. One-tail hypothesis test H 0 : The mean survey rating of female students is higher than or equal to the male students (μ1 ≥ μ2). H A : The mean survey rating of female students is lower than the male students (μ1 ˂ μ2). Significance level (α)= 0.05 The result of hypothesis testing shows that: there are significant differences between male and female students' impression about the camp activities in survey question 1 and survey question 16. The onetail hypothesis test indicates that: the male students rated the survey question 3,5,11, higher than female students. These results can be interpreted as: the male students thought the game based learning camp activities were more enjoyable and interesting than female students.
To test the racial difference, we performed both two-tail and one-tail hypothesis test for Caucasian and non-Caucasian participants. The hypothesis for "Qi. I enjoyed learning about computer science" was: 1. Two-tail hypothesis test H 0 : There is no difference in mean survey rating of Qi between Caucasian and non-Caucasian students (μ1=μ2).
H A : There is difference in mean survey rating of Qi between Caucasian and non-Caucasian students (μ1≠μ2). Significance level (α)=0.05 2. One-tail hypothesis test H 0 : The mean survey rating of non-Caucasian students is higher than or equal to the Caucasian students (μ1≥μ2). The result of hypothesis testing between racial group shows that: there are no significant differences between Caucasian and non-Caucasian students' about the camp activities and experiences. The one-tail hypothesis test indicates that the Caucasian students rated the survey question 11 higher than non-Caucasian students. This result shows that the Caucasian students have higher self-efficacy than non-Caucasian students after the game-based learning camp activities.

CONCLUSION
This paper describes evaluation results of game based learning for high school cybersecurity education. Four cybersecurity computer games were developed to educate social engineering, secure online behaviors, and 10 cybersecurity first principles. The use of game based learning in the PNW GenCyber camp was an excellent platform to teach cybersecurity principles and secure online behaviors for high school students. This approach is beneficial to the future cybersecurity workforce by exposing more high school students to the cybersecurity education pathway at a time when they are making decisions regarding higher education. The game based learning method was well received by the students, support staff, instructors, and site visit team. This was also supported by the post camp survey conducted for 154 participants with average rating of 4.26 out of 5. To investigate the gender and racial difference of game-based learning, hypothesis testing for two population means with unequal variance (T-Test) was conducted to test gender and racial difference. The hypothesis testing result indicated that the male students thought the game based learning activities were more enjoyable and interesting than female students, and the Caucasian students showed higher self-efficacy than non-Caucasian students after the game based learning camp activities.