Evaluating the Effectiveness of an Online Simulation to Teach Business Skills Abstract

Evaluating the Effectiveness of an Online Simulation to

Teach Business Skills

Nicola Whitton

Manchester Metropolitan University

Dr Niki Hynes

Napier University Business School

Abstract

Educational computer games and simulations are increasingly being seen as a novel way to facilitate student interaction and engagement. However it is often unclear exactly which aspects of game-based courses students are motivated by or how they interact with each other on such courses. This paper describes the implementation and evaluation of a collaborative game-based course for final-year marketing students and discusses student expectations, motivations, attitudes toward the game and group working patterns.

Introduction

Games and simulations have been used in the field of education for many years, particularly in the areas of business, training staff in financial and economic skills, combat training and war gaming. The health sector commonly uses similar realism techniques to those in games and aircraft pilots, and other vehicle drivers, often use simulations in the early stages of training (Kirriemuir, 2002). Internet-based games are also commonly used the areas of education, business and policy (Askawa, T. & Gilbert, 2003) to provide a safe but realistic experience of the real world.

Constructivism is now considered by many to be the dominant teaching and learning paradigm (Cooper, 1993) and in recent years, the focus of online learning environments has been on developing constructivist learning environments and to facilitate authentic and problem-based learning (e.g. Grabinger et al., 1997; Jonnassen, 2002). The basis of a constructivist learning includes situated learning, active learning and social construction (Savery & Duffy, 1995) and has much in common with the design of experiential multi-user online games. Educational computer games and simulations are increasingly being seen as a novel way to increase student engagement with a topic while encouraging experiential learning and facilitating collaboration (e.g. Prensky, 2001).

However, one of the main reasons often cited by researchers for the potential value of educational games is the positive motivation of students to play them. Although it cannot be argued that many students do appear to find games motivational and engaging, the assumption is often made in the literature on game-based learning that this is true of all students (see, for example, Alessi and Trollip, 2001; McFarlane et al., 2002). Much of the evidence given to support this assumption is experiential and anecdotal and fewer learners,

2Vol. 9 No. 1, March 2006 - Whitton & Hynes

particularly adults, than might be expected are actually positively motivated by games (Beasley & Crerar, 2004).

In addition to the educational value of a game itself, the way in which it is integrated as part of a course can influence its educational effectiveness and levels of student acceptance and engagement. This paper describes the implementation of a collaborative, final-year undergraduate marketing course at Napier University in Edinburgh, based around the use of the online business simulation game, Marketplace. Marketplace is a virtual competitive business environment, in which groups of students work as companies analysing markets, designing their marketing strategies, and developing and marketing products. The game aims to teach a range of hands-on managerial skills within a realistic environment.

The use of the Marketplace game within the course was evaluated, looking particularly at student motivations to learn from games, their expectations from using educational games, their perceptions of the computer game environment, their group working, decision-making and communication patterns and their overall experience of the course. The results of this evaluation are presented here and the implications for future iterations of the course and for the implantation of game-based courses in general, are discussed.

Implementation

The Marketplace online business game was used as the basis of a final year undergraduate marketing course, Marketing Management in Practice, at Napier University in Edinburgh. Marketplace is an Internet-based simulation game in which groups of students run virtual companies and compete against each other for market share and position. The groups have to make decisions about product design, finance, advertising, promotion and sales force expenditure. Marketplace aims to provide hands-on experience of the business environment and provide an overview of all the processes within the organisation, within a realistic, intuitive, easy-to-use learning environment (Cadotte, 2003).

Students were split into teams, or ‘virtual companies’ of four or five members and were expected to work within those teams for the duration of the course. The course was fifteen-weeks in total and the game was broken down into eight decision periods, each corresponding approximately to a week in real time. During each decision period groups were expected to make certain decisions about the company, its strategies, and products. Although Marketplace is an online game and decisions are input online, students were expected to meet face-to-face to discuss decisions before one individual from the team updated their company profile online at the end of each decision period.

Face-to-face classes were split between lectures and tutorials, with five lectures in total and a three-hour tutorial slot each week in which students were expected to make their group decisions, although they were also able to meet outside the tutorial slot. Attendance at the tutorial was made compulsory by including a marked tutorial worksheet that had to be signed by all group members and handed in at the end of each tutorial. At the end of each decision period, the simulation game provided immediate feedback and showed the performance of companies relative to one another within the virtual environment.

Although the performance of each team was measured against that of other teams to provide a competitive environment, students were not assessed on the performance of their team. There were three forms of assessment, split in emphasis between group and individual performance: a presentation to the ‘Board of Directors’ about performance to date and future

Evaluating the Effectiveness of an Online Simulation to Teach Business Skills 3

plans after quarter five of the game, worth 40% of the final mark; individual assignments at the end of the simulation, again worth 40%; and team work sheets detailing decisions made in each tutorial period, worth 20%.

Marketing Management in Practice using Marketplace was piloted during the academic session 2004/5 and 42 students elected to study on the module.

Evaluation methodology

As the Marketing Management in Practice module was run for the first time based around the Marketplace game, it was felt to be very important that an independent evaluation of the effectiveness of the game-based module was carried out. A teaching research grant was secured from the Higher Education Academy Business, Management and Accountancy subject centre, which enabled this to happen.

Before the start of the module, the 42 students who had elected to study the module were asked to complete an attitudinal questionnaire. This covered the areas of computer game playing, motivations for game playing and attitudes towards educational games as well as background questions regarding age, sex and experience with games. The purpose of this initial questionnaire was to provide baseline information on the students taking part in the module and 41 students completed this questionnaire.

For the main evaluation, a range of qualitative and quantitive measures were used to illicit information about the student experience of learning in groups with an educational game. The module was evaluated in three ways: six focus groups took place half way through the module; students were asked to complete an attitudinal questionnaire at the end of the module; and a reflective statement on their learning experiences within the game was mandatory as part of the final assessment of the module.

The focus groups took place between weeks six and eight of the module and aimed to explore the range of student experience on the module. In total 20 students took part in the focus groups and although there may be positive bias because the focus group was optional, students taking part were paid a small fee in order to encourage a range of opinion and attitudes with the groups. Each focus group had three or four students in it and took approximately one hour. Each followed the same structure but questions were open-ended and the interviews were used as an opportunity for the groups to explore their experiences in any direction that seemed appropriate. The use of focus groups over individual interviews was chosen partly because of time constraints and partly because it was hoped that a group interview would help to stimulate ideas and debate. The structure of the focus groups aimed to avoid interviewing students who were in the same work groups together as it was felt that this might hinder honest discussion of experiences.

The focus groups examined the expectations of the students who elected to study on the course and their motivations for doing so, patterns of working and methods of communication, both on- and off-line, how the groups allocated responsibilities and made decisions, what they were learning on the course, the assessment and the overall module and game design. Data from the focus groups were used to isolate themes and ranges of opinion for further study, particularly to design an attitudinal questionnaire that was presented to all students at the end of the module as a way of quantifying some of the opinions that had been highlighted. This questionnaire examined reasons why students studied the module, perceived learning outcomes, communication styles and attitudes to game playing and team working.

4Vol. 9 No. 1, March 2006 - Whitton & Hynes

This questionnaire was presented to students in the final week of the course and in total 26 students completed and returned it.

In addition, as part of the final individual assessment for the module, students were asked to complete a short reflective statement on their experiences playing the game and working collaboratively with others. This reflective statement was used partly as a tool for debriefing students from the gaming experience, a task that is thought by many to be as important as the game itself (Thiagarajan, 1993), and partly to provide another source of qualitative data on the student experience of the module.

Although the evaluation in total examined a range of areas, those described in this paper focus on student attitudes towards educational games, motivations for game playing, their expectations from playing the game and reasons for taking part in the module, their attitudes towards a computerised game and group working and communication patterns both on and offline.

Results

In this section the results of the evaluation described above are presented and examined. Three main areas are highlighted as being of interest here: the expectations of the students taking part and their motivations for doing so; their attitudes towards using a game for learning and implications of the fact that the game used was computer-based; and issues associated with teamwork, decision-making and communication patterns.

The students who took part in the module were final year students from a range of marketing and business disciplines. They were predominantly aged in their early twenties (90% aged between 21 and 25) and approximately two-thirds of the class were female. Most of the class had previous experience of playing computer games (only 7% said that they had not played one before) and although only 12% said that they play computer games regularly now, an additional 50% do so occasionally.

However, despite the large percentages of students who had experience playing computer games, only 12% had used an educational game of any sort before. With this group of young people with experience of game-playing it might be expected that the use of an educational game would be strongly motivational, however the vast majority of students (85%) said that using a game to learn would not motivate them either way, positive or negative. This is particularly interesting as the group of students who took part in this module self-selected to take part and could be assumed to be positively disposed towards games.

As a positive indication of the overall success of the module, 80% of students who completed the final questionnaire said that they had enjoyed this module more than other modules. While this data can only be indicative as only 60% of the class completed the final questionnaire, it provides some evidence that the module was well received by the students.

Expectations and motivations

The students who took part on the module did not seem to do so primarily because the module was based around a game, so it was thought to be important to explore the range of motivations and expectations of the students why studied the module.

Evaluating the Effectiveness of an Online Simulation to Teach Business Skills 5

During the focus groups a number of reasons for electing to study on the module were brought forward including previous experience of similar games, recommendations from friends and the fact that an online game would produce immediate feedback. The questionnaire was used to examine which of the motivations described in the focus groups were most prevalent among the students on the module.

Four primary reasons for taking the module emerged that appeared to influence more of the students than others. The most popular reason for taking the module was that it was new and different (92%) so novelty appeared to be a primary motivational factor. Second was the fact that the module sounded fun and interesting (72%). More pragmatically, 69% of students cited the fact that the module was continually assessed with no examination as one of their key reasons for electing to study it and 58% said that they wanted to do the module because it provides real life experience of all the aspects of a business.

When asked if they were motivated to do well in the module, 96% of the students said that they were. In contrast, 77% said that they were motivated to do well in the game. As performance in the game did not form part of the final assessment, this is not contradictory. From the focus groups, it appeared that motivation to win the game changed throughout the life of the game depending on the performance of the group that the individual was in, with winning groups becoming more motivated to win while groups who were performing more poorly quickly lost their motivation.

There was a large amount of competition between groups, but not among individual members of groups and this appeared to be a positive motivation, but again more so for teams that were performing better. Other factors that were cited as contributing to positive motivation were the fact that feedback from the game was immediate, the fact that the game itself was highly engaging and immersive, and the enthusiasm and motivation of the lecturer was also motivational for students.

Attitudes towards a computer-based game

During the focus groups it emerged that some of the students had issues with the fact that the game was computer-based. Positive factors included that feedback from each decision was provided immediately at the end of the decision period, however it was also felt that the feedback provided by the game was limited and that it does not provide enough information on exactly why certain actions led to specific consequences.

The game was also criticised in the focus groups in that it did not seem to reflect the application of marketing theory and that it would be possible to win by making random decisions. However, from the questionnaire results only 7% of the students thought that it would be possible to win the game using guesswork. Because of the computerised nature of the game, the majority of students perceived it to be fair and objective (65%) but another criticism highlighted in the focus groups was that the game was based on a very limited model with limited options and that there was no potential within the game for creativity or expression of individual talent. This was particularly an issue when devising advertising materials based on a list of criteria. However, from the questionnaire only 34% of students said that the game options were limited and only 27% thought that a more complicated game would be more enjoyable.

The nature of the game, being computer-based, also meant that there was a perception that there was a single way to win based upon a specific strategy, again reducing the potential for creative flair and the development of niche markets and for innovatively differentiating

6Vol. 9 No. 1, March 2006 - Whitton & Hynes

between products. Also the nature of the products developed in the game – computer systems – were seen by some of the less computer-literate students as providing an unfair advantage for students who knew more about the product.

Teamwork and communication patterns

The students were also asked about the ways in which they worked together as teams and how they communicated together both on- and off-line. From the focus groups, the students said that the module was more time-consuming than other modules, although they did not seem to resent the time spent as it was perceived by the majority as being worthwhile. In the questionnaire 42% said that they spent more time on this module than other modules. Reasons for the extra time spent centred around the fact that students wished to take the time to understand all of the aspects of the business. Students were asked to allocate roles and concentrate on certain individual areas, but in practice only 42% of students did this and the majority made all decisions about all areas of the business together, with 88% of students saying that all decisions were made as a group. Students cited an understanding of the interactions between the different business areas and gaining an overview of how decisions affected different areas of the business as one of the important learning outcomes so they wished to be involved in all decisions and understand all aspects of the business, which was inevitably time-consuming. While this enthusiasm to learn about all aspects of the game is not necessarily a bad thing it could be argued that it shows a lack of strategic ability by the students to make the best use of the team resources to increase the effectiveness of the group.The aspect of teamwork were viewed favourably, with 92% of students saying that they enjoyed working in their team and 88% saying that they would like to work in the same team again. Tutorial attendance was compulsory as attendance was assessed but this was viewed extremely positively by the students, with only 11% saying that they disliked having to attend every tutorial.

Although students met face-to-face in tutorials, there were a number of other communication options open to them including email, telephone text messaging, telephone and online messenger. All of these options were used by some of the groups to some degree, with mobile phone messaging (35%) and email (31%) being the most regularly used after face-to-face meetings (92%). Messenger software was least commonly used with 50% of students never using it to communicate with their group, but it was used regularly by groups where the members spoke different languages as their first language or when one group member could not make a meeting as it enabled them to take part in the decision-making from another location.

Discussion and implications

In general, the students appeared to find the course a positive experience. However, the students that undertook the course were self-selecting and so are likely to be those that, if not positively motivated by playing games, are at least not negatively motivated. The students appeared to find greater motivation from the novelty factor than any other and it could well be the case that if they were faced with a number of modules in this format any motivational factors caused by the design of the course would quickly disappear.

Evaluating the Effectiveness of an Online Simulation to Teach Business Skills 7

The students were also positively motivated by the intra-group competition – particularly when they felt that they could win. It is important however to be careful when using a game such as this that the situation does not become overly competitive to the detriment of learning. Separating the assessment from the game performance was one way of achieving this so that the final mark of one group or individual could not affect the final marks of others, whereas in the game all group performances are interrelated.

The relatively young population that took part in this module had much experience of computer game playing and were not put off by the game-based format of the module, but with an older, less computer-game-literate population this may be more of an issue. The students felt that the game model was fair but somewhat restrictive and provided limited options and feedback. Setting the game in a wider context and incorporating additional elements (e.g. production of physical advertising materials) would help to add more variety to the game, increase the opportunities for creativity and provide more scope for personal feedback.

The students seemed to enjoy working in teams but did not always appear to be making the most efficient use of their time as part of a team, but instead most team members wanted to be part of all decisions. The module did not contain any content on how to work efficiently as a team, how to make group decisions and alternative appropriate methods of communication at different times. It is felt that adding these elements at the beginning of the module would help students to gain more from the team experience.

In terms of the design of the evaluation it should also be noted that the sample size in this study was small and the response rate from the second questionnaire was also relatively low and therefore may show bias. However, by the use of a number of different methods of evaluation the evaluation aimed to provide a more rounded overview of results. In future years, it is hoped to run the module with larger numbers of students and to complete a further larger-scale evaluation.

From this evaluation it is possible to draw together a number of implications for future iterations of the module, and for using computer games within teaching in general. These include:•

Do not necessarily rely totally on the structure provided by the game, but add in additional creative elements that enable students to innovatively differentiate themselves from other groups, and allow space for out-of-game discussion and debriefing.

Consider ways of encouraging intra-group competition as a motivator but be wary of creating an overly competitive environment that may detract from the learning outcomes of the module.

Do not assess game performance as the relationship between positive performance and positive motivation may be disproportionally detrimental to those who do not perform well during the game.

Provide information to students, particularly those who have limited experience of team working, on the most effective ways to work, communicate and make decisions as part of a team so that they are aware of the group processes that take place and can make a conscious decision to use their time most effectively.

•

•

•

8Vol. 9 No. 1, March 2006 - Whitton & Hynes

References

Alessi, S. M. & Trollip, S. R. (2001) Multimedia for Learning. Boston: Allyn and Bacon.Asakawa, T. & Gilbert, N. (2003) Synthesizing experiences: lessons to be learned from internet-mediated simulation games. Simulation and Gaming, 34/1, 10–22.

Beasley, N. & Crerar, A. (2004) Games, motivation and adult learners. In Proceedings of ISAGA 2004, Bridging the Gap: Transforming Knowledge into Action through Gaming and Simulation. Munich, Germany.

Cooper, P. A. (1993). Paradigm shifts in designed instruction: from behaviorism to cognitivism to constructivism. Educational Technology 33/5 12–19.

Cadotte, E. R. (2003) What is Marketplace? Marketplace Business Simulations [Online] http://www.marketplace-simulation.com/what-is-the-marketplace.php

Grabinger, S. J., Dunlap, J. & Duffield, J. (1997). Rich environments for active learning. ALT-J, 5/2, 5–17.

Jonnassen, D. H. (2002). Engaging and supporting problem solving in online learning. The Quarterly Review of Distance Education, 3/1, 1–13.

Kirriemuir, J. (2002) Video Gaming and Digital Learning Technologies: Relevance and Opportunities. Delib Magazine, February.

McFarlane, A., Sparrowhawk, A. & Heald, Y. (2002) Report on the educational use of games. TEEM. [Online] http://www.teem.org.uk/publications/teem_gamesined_full.pdfPrensky, M. (2001) Digital Game-based Learning. New York: McGraw Hill.

Savery, J. R. & Duffy, T. M. (1995). Problem-based learning: an instructional model and its constructivist framework. Educational Technology, 35, 31–38.

Thiagarajan, S. (1993). How to maximise transfer from simulation games through systematic debriefing. In S. Lodge, F. Percival and D. Saunders (Eds) The Simulation and Gaming Yearbook, Volume 1: Developing transferable skills in education and training. London: Kogan Page.