All-in-One: Impact study of an online math game for educational purposes

The purpose of this evaluation study is to address the impact of an educational game for strengthening mathematical skills and knowledge. Within this framework the study deals with the appr opriateness of specific game elements corresponding to the success of a mat h game taking 65 Austrian players and a control group of 41 players of a different game into account. While an exploratory factor analysis confi rms the validity and reliability of the 30-item scale, linear regression a alyses illustrate the significant positive impact of the game's flow to i ts success. An independent ttest as well as an ANOVA demonstrate no significant differences between genders. However, a Kruskal-Wallis test stresses th at the game elements are significantly important for different age-groups. I n addition, the results show that the game elements should be chosen carefully w ith respect to the educational objectives. This impact study supports the positive research stream referring to the high potential of games as an educational instrument in math education.

Although only few authors have developed game for educational purposes and tested their impact (e.g.McDonald and Hannafin, 2003;Ando and Harrington, 2006;Zyda, 2007), in particular a math game (Vogel et al., 2006), a review of 30 online games dealing with mathematics has already analyzed different changes in motivation and cognition when playing these specific games (Erickson, 2015).For instance, McDonald and Hannafin (2003) focused on a television game show called "Who Wants to Be a Millionaire" for exam preparation.Another researcher stressed a game that focused on prevocational math education to improve students' skills in proportional reasoning (ter Vrugte et al., 2015).A further contribution comparing the effects of a math game concludes that game playing can improve cognitive abilities such as multitasking and visual short-memory as well as spatial skills (Núñez Castellar et al., 2015).In general, a growing body of research documents the positive effects of game-based learning instruments on attitudes (e.g.Garris et al., 2002;Bogost, 2007;Connolly et al., 2008;Fellnhofer, in press;in press) as well as on performance in mathematical skills acquisition in the long run (Garris et al., 2002;Backlund et al., 2008;Erickson, 2015) such as arithmetic performance (Núñez Castellar et al., 2015).
Overall, there is great potential (Dede, 2000;Erickson, 2015) -in particular, at an international level, the latest findings of Zhang (2015) show that about 10 million online surfers spend roughly 3.71 million days per year on a popular online math game for children.
However, some authors disagree with this positive development (Wouters et al., 2009;de Freitas and Ketelhut, 2014) and state that there is not enough evidence that games meet the criteria for efficient pedagogical tools (Zhang, 2015).This calls for more research with different approaches stressing, the encouraging achievements of these pedagogical inventions.In line with this intention, this contribution will focus on different attributes of games to affect both learning performance as well as game-playing motivation, taking different age-groups and genders into account.To assess the skills stabilization in math education, the author has been developing a web-based math game to answer the following research question: Does a simple math game in a farm setting characterized by multiple-choice questions have an impact on learning success?
This research question will be addressed as follows: A theoretical framework and hypotheses development will build a solid basis for both the methodological and the results sections in which the outcomes of the hypotheses will be presented.These will be critically discussed in the final section dealing with limitations and implications.

Theoretical Framework and Hypotheses
Growing acceptance for the entertaining aspects of educational games (Rosas et al., 2003;Virvou et al., 2005;Paraskeva et al., 2010) -so called "edutainment" (Okan, 2003) -increases the learner's motivation, as has been acknowledged by famous digital learning game designers such as Prensky (2001).While games with educational objectives are being implemented across a variety of disciplines (Connolly et al., 2012), adequate games focusing on mathematics are relatively rare (e.g.Nte and Stephens, 2008;Wijers et al., 2008;Connolly et al., 2012).Although an increasing number of academics have been dedicating their attention to the relationships between math game players and improved perceptual, cognitive, behavioral, and affective performance (ter Vrugte et al., 2015;Zhang, 2015;Sinopoli et al., 2015;Núñez Castellar et al., 2015), little is known about the motivational factors of successful online games with a focus on education.Essential game characteristics that motivate and facilitate the process of learning, for instance indicators of challenge and skills (based on Qin et al., 2009), curiosity (based on Qin et al., 2009), flow (based on Wu et al., 2013), and enjoyment (based on Fang et al., 2010), require in-depth understanding from all parties involved in the phases of game development (Charsky, 2010).However, not all elements or features of a math game will foster the game's effectiveness (Wouters et al., 2009;Connolly et al., 2012).In addition, complexity of game design is not seen as a guarantee for success; in other words, when complex educational content is transformed into simple games, potential is born (e.g.Salen and Zimmerman, 2004;Higuchi et al., 2005;MacCalla et al., 2015).The generalization of previous research outcomes to a variety of game-based settings dedicated to mathematics is insufficient.Based on previous evaluations of the effectiveness of educational games (e.g.Núñez Castellar et al., 2014;Fellnhofer, 2014;2015;and in press), we propose the following: Hypothesis 1: Different game elements support the success of a math game differently.
In general, mathematical games with educational objectives have to be designed with respect to pedagogical issues in order to have a positive impact on learners (Rosas et al., 2003;Divjak and Tomić, 2011).Prior studies have investigated the effects of age upon attitudes towards online learning, concluding that age makes a significant difference (Comber et al., 1997).Scholars target different age-groups in their studies (Ratan and Ritterfeld, 2009).The majority of studies conducted on math games have dealt with primary school; secondary school constitutes the next-largest group (Divjak and Tomić, 2011).Schrader and McCreery (2008) examined differences in age-groups, concluding that the younger the participants, the higher their level of expertise in online games.Based on these findings, we assume Hypothesis 2a: The game satisfies different age-groups differently.
Furthermore, Terlecki and Newcombe (2005) state that games tend to be gender-oriented.Moreover, Feng et al. (2007) stress that female players show less experience in playing than their male counterparts, which might influence the game's benefit.Concerning gender differences, another study came to the conclusion that different genders prefer different game types (Lucas and Sherry, 2004) and consequently behave differently when playing the game (Eastin, 2006).In addition, Chou and Tsai (2007) argue that males are willing to spend more resources on playing games than females, and that males tend to enjoy games significantly more.Although scholars do not show strong agreement on the discussion of gender differences when it comes to educational games, few researchers found no significant differences between the genders (e.g.Darken et al., 1998;Papastergiou, 2009).Thus, this research suggests that Hypothesis 2b: The game satisfies different genders differently.
Overall, games for educational purposes are developed to encourage the acquisition of knowledge (Lindh et al., 2008).Previous studies used instruments looking at game preferences such as success items or hours played (Lucas and Sherry, 2004;Beale et al., 2007;Boot et al., 2008).A significant difference was found between educational game players and those that had not played any games for learning (Miller and Hegelheimer, 2006;Beale et al., 2007;Yip and Kwan, 2006;Fellnhofer, in press).Furthermore, based on previous results showing that different effects occur based on different learning objectives (e.g.Cameron and Dwyer, 2005), we assume that game elements need to be adapted to the intended learning outcomes or subject.Thus, in line with this argumentation, the following is proposed: Hypothesis 3: The game elements have to be adapted to the teaching subject.
The theoretical framework, the hypotheses, and the assumed directions are illustrated in Figure 1.

Sample characteristics and data
The data of this research sample consisting of 65 students were collected in Austria in different workshops from May to July 2015.While Table 1 illustrates the descriptive statistics of the survey sample, Table 2 outlines the control group -experts questioned regarding the suitability of the game for to entrepreneurial education -for testing hypothesis 3, whether game elements need to be adapted to the educational objectives.

Measurement
The self-developed scale to measure game success consists of eight items and focuses on motivation and interest in the game, similar to previous similar investigations (Darken et al., 1998;Lindh et al., 2008;Connolly et al., 2012).Indicators of enjoyment are based on Fang et al. (2010), measured with eight items.Indicators of flow consist of five items which have been adopted from Wu et al. (2013).Both indicators of curiosity are modified from Qin et al. (2009), measured on a six-item scale, as are indicators of challenge and skills with three items.Based on these previously published articles, the translated items -from English into German -were measured on five-point Likert-type scales (1=''strongly disagree'' to 5 =''strongly agree'').

Game
The math game's storyline is straightforward.The game player or 'farmer' receives monetary remuneration and livestock for correct answers to mathematical questions while competing with other gamers in the setting of a small farm village.This math game in a farm setting has already run through Experienced game player yes 35 no 30

Sum of math game players 65
several assessments to increase the quality of different criteria for an efficient pedagogical product.First of all, a qualitative assessment stresses the adequately implemented design based on recommendations and results of prior studies (e.g., Schulmeister, 1997;Kerres, 2001;Van Merrienboer and Sweller, 2005;Fellnhofer, 2014).Being a funded project, the game has already passed several peer-reviewed evaluation project phases which have led to further improvements in aspects such as content, structure, storyline, and design.Overall, this game meets the necessary requirements for math games that aim to increase motivation by making learning mathematics easier based on prior research (Divjak and Tomić, 2011) in the following areas: a) different subjects based on the curriculum; b) order of activities for learning; c) a basic learning model; d) content presentation; e) interface including words and objects; e) appropriate navigation structure; f) feedback and reward system; and g) fun elements such as graphics, sound, story, characters, and humor.

Analysis
First of all, an exploratory factor analysis (EFA) documents the reliability and validity of the construct, stressing an adequate overall model fit through the following: a) almost all communalities of all items are greater than 0.5 (Field, 2005); b) all Cronbach's alpha are greater than 0.7 (Nunnally, 1978); c) all Kaiser-Meyer-Olkin Measures of Sampling Adequacy are more than 0.5 (Kaiser, 1974); d) all determinants of the constructs' correlation matrix are greater than the necessary value of 0,00001; and e) all significant values conclude that there are correlations in the data set that are suitable (Bartlett, 1937).Accordingly, multicollinearity among these non-critical variables can be accepted (Farrar and Glauber, 1967).Appendix 1 presents the results of the EFA.Next, stepwise linear regressions and Pearson's Bivariate Correlation analyses were performed to test hypothesis 1.Previous impact studies for the purpose of learning game assessment applied regressions (e.g.Weibel et al., 2008) and correlation analyses too (e.g.Jennett et al., 2008;Weibel et al., 2008;Castelli et al., 2008;Fu et al., 2009).As indicated in Table 3, the Durbin-Watson for hypothesis 1 is between 1.5 and 2.5.Thus, it can be assumed that there is no first-order linear auto-correlation in the data.Because of single informants for independent methods, common-method bias is crucial in this study.To respond to the issue of common-method variance, three approaches have been taken (Podsakoff et al., 2003): the dependent variable is constructed by means of information from sources different than those of the independent variables; the order of the questions is mixed; different scale types were used; and complicated specifications of regression models are implemented.Finally, common-method bias does not appear to be a critical concern in this reading.

Results
Table 4 outlines (stepwise) the results of the linear regression analyses for testing hypotheses 1 and 2. Indicators of flow show a significantly positive influence (β=0.782***)regarding the success of the game.Additionally, as presented in the Pearson's Bivariate Correlation Matrix (Appendix 2), the game's success is correlated to the indicators of flow (0.784**), indicators of curiosity (0.739**), indicators of challenge and skills (0.450**), and indicators of proactiveness (0.513**).Thus, hypothesis 1 can be rejected and we stress that game elements support a math game's success.In particular, it has been discovered that flow is positively associated with a learning game's success.Furthermore, we scrutinized the impact regarding the intensity of playing.A positive influence was assumed and supported by the linear regression analyses (β=0.275***), as presented in Table 4.In addition, as presented in Appendix 2, the correlation matrix shows positive relations between hours played and the game characteristics such as flow (0.346**), curiosity (0.412**), challenge and skills (0.303*), and proactiveness (0.262*), as well as game success (0.443**) in general.Consequently, the more hours the game is played, the higher its positive impact.The excluded variables within the results of the linear regression analyses (stepwise) indicate that gender and age do not have any influence.For hypothesis 2 an independent t-test and Kruskal-Wallis test as well as an ANOVA provide further significant results.These tests confirmed the detected equality between genders.However, as shown in Table 6, significant differences were found between the age-groups within a Kruskal-Wallis test.While agreement occurs when it comes to rating the success of the game and importance of indicators of enjoyment, players from different age-groups are significantly different in their perspective regarding indicators of flow (0.038**), curiosity (0.012**), challenge and skills (0.008***), and proactiveness (0.006***).Therefore, it can be concluded that the game appears not to be adequate to all different age-groups.As indicated in Table 7, for hypothesis 3 an independent t-test comparing the points of view of 65 math players and 41 experts regarding on the game dedicated to entrepreneurial education provided significant results for all game elements with the exception of flow.These results support for hypothesis 3, that the game elements have to be adapted to the teaching subject.Finally, as presented in Figure 2, while positive support for hypotheses 1 and 3 has been found, the data do not reflect enough evidence for gender differences.Nevertheless, age and the educational topic do play a significant role in game design, and therefore should be taken into account for the sake of the game's success.

Discussion
Despite the fact that educational games are widely used these days, from a literary point of view their efficiency with regard to motivation and learning success has not yet received strong support (Maertens et al., 2014).Fundamentally, however, researchers share the opinion that educational games have great potential (e.g.Núñez Castellar et al., 2015), which should not be neglected either in the classroom or in academic discussion.Despite the growing interest and positive agreement, there is comparatively little research regarding the essential game characteristics required to increase the learner's motivation for games dedicated to mathematics.This notion forms the basis of this research, relying on a solid groundwork of literature on both successful game design (Qin et al., 2009;Fang et al., 2010;Wu et al., 2013) and webbased math education (Erickson, 2015).Moreover, it enhances previous studies (e.g.Divjak and Tomić, 2011) with a detailed insight into parameters of successful games that will positively influence the learner's outcome.Further, while gender issues do not appear to crucial in this framework, light has to be shed on the factors of age and educational objectives.Accordingly, based on our findings, it can be concluded that specific game elements support diverse age-groups differently.However, in line with previous findings, our research supports the idea that implementing mathematical games in the curriculum increases the efficiency of educational achievements at all levels of education (Divjak and Tomić, 2011).

Limitation and further research
Like every other study, we must review the limitations of our investigation.It implies that game characteristics such as flow, curiosity, challenge and skills, and proactiveness, as well as age serve as important triggers or antecedents of game success with respect to the intensity of playing, and thus have to be included in future research.While these results are limited to one math game, further research calls for extensive comparisons between different math games at different levels.Does flow significantly influence the success of other math games as well?
Despite a control group of 41 experts, this research is based on a relatively small sample (65) with a geographical focus on Austria.As a consequence, these limitations call for further international research.However, similar contributions also rely on a comparably small sample size (e.g.Darken et al., 1998 with 40;Wee, 2004 with 52;Nelson et al., 2006 with 62;Friedrich et al., 2006 with 27;Radu and Loué, 2008 with 44;Nabi et al., 2008 with 50;Cheung, 2008 with 50; Singh and Verma, 2010 with 52).In addition, there is a great need for further longitudinal or multi-periodical research.Backed by these results, further investigations are required for different age-groups.While more qualitative investigations will provide a better understanding of the nature of the game characteristics involved (Lindh et al., 2008), further quantitative research is required to present a more solid identification of the different interplays of these variables.The multidimensional interconnections in game play for educational purposes have been strengthened through specific game elements such as flow in this research.Nonetheless, further research is required to analyze the game characteristics-performance relationship in more detail, with a particular focus on independent, mediating, and moderating, as well as interaction effects.

Conclusion
From a theoretical perspective the crucial concept-building potential of educational games as a solid pedagogical instrument is here supported.In essence, in line with prior investigations (e.g.Núñez Castellar et al., 2014), this contribution considers game play an effective facilitator or motivational tool for improving math skills.While further research is required, the results contribute to raising awareness regarding motivational factors of different math games for different age-groups.In particular, the analysis promotes ageadequate learning by game-based technology and development, whereby the elementariness of game flow has the potential to foster learning.Finally, from a practical perspective, this contribution emphasizes that game-playing can be used to assist human resource departments to assess and train math skills of (potential) staff relevant for specific jobs (e.g.accounting, etc.).Overall, the paper contributes to the emerging literature on educational games with a central approach on the multilevel interplay, which is instrumental for embedding games as solid pedagogical instruments in education.

Figure 2 :
Figure 2: Results of the research model

Table 1 :
Sample characteristics

Table 2 :
Control group

Table 3 :
Fit statistics

Table 6 :
Statistical results of Kruskal-Wallis test

Table 7 :
Independent Samples Test