Business intelligence serious game participatory development: lessons from ERPsim for big data

Abstract

Purpose

A major trend in enterprise resource planning software (ERP) is to embed business analytics tools within user-centered roles in enterprise software. This integration allows business users to get better and faster insight to action. As a consequence, it is imperative for business students to learn how to use these new tools to adequately prepare them for new expectations in the industry. The paper aims to discuss these issues.

Design/methodology/approach

In this paper, the authors propose a new serious game, called ERPsim for big data, to enable the learner to acquire abilities at each level of the business analytics learning taxonomy. To maximize the pedagogical impact of the game, participatory design (PD) with professors as co-designers was used during game development.

Findings

This case study presents the PD approach and analyses the efficacy of the proposed new simulation.

Originality/value

The authors conclude by providing recommendations and lessons learned from this approach.

Citation

Labonte-LeMoyne, E., Leger, P.-M., Robert, J., Babin, G., Charland, P. and Michon, J.-F. (2017), "Business intelligence serious game participatory development: lessons from ERPsim for big data", Business Process Management Journal, Vol. 23 No. 3, pp. 493-505. https://doi.org/10.1108/BPMJ-12-2015-0177

1. Introduction

Building on new advances in business analytics, forward looking organizations are setting up technological infrastructure to leverage the potential of what is been referred to as big data (vom Brocke et al., 2014). However, there is a shortage of skilled workers who have the expertise necessary to optimize the use of those tools (Wixom et al., 2014). While there has been an important increase in students registering in big data classes at universities and colleges (Davis and Woratschek, 2015), new pedagogical approaches are needed to help develop the skills and attitudes toward business intelligence required of the future workforce (Davenport and Patil, 2012).

This paper presents a case study of the collaborative efforts deployed to develop an enactive approach to teach business intelligence concepts and to provide hands-on experience to students in an authentic and dynamic business environment. Using participatory design (PD), a well-known approach at leveraging the collaboration of designers and users to co-innovate in the development of an IT artifact (Halskov and Hansen, 2015), we developed a business simulation game that allows players to experience the challenges of big data. This new serious game was developed upon previous work on a simulation called ERPsim (Léger, 2006; Léger et al., 2007). Through the business simulation, participants learn to use an enterprise resource planning (ERP) system, to collaborate as a team, to understand the business environment and to make and implement business decisions. Until now, ERPsim was mostly aimed at learning goals related to operational excellence (Léger et al., 2012), the focus on big data analytics was limited. This new version of ERPsim intends to bring big data learning goals at the forefront of the experience.

In this paper, we report our approach and discuss the appropriateness of using a PD development method in this context. This co-design approach has led to the development of ERPsim for big data, a new version of our simulation that, results suggest, provides analytical abilities that expand the competency of enterprise system learners. We also discuss the effectiveness of the proposed simulation with faculty users involved in the beta testing phase. We conclude by identifying the lessons learned and provide recommendations for PD to develop enactive pedagogical material to teach IT as a means to identify pedagogical needs and preferences of an academic community.

ERPsim is a serious game developed for business school students to learn business process management using an ERP system (Léger, 2006; Léger et al., 2007). In the ERPsim simulation, teams of students interact through a real SAP system. Teams compete in the same virtual market for customers and products. For example, each team chooses to order certain amounts of raw materials to create a product, to ship quantities of that product to different regions and to set the price in each region. According to these decisions, each team makes a profit and the team with the highest profits wins the simulation. Through the game, participants learn to use an ERP system, to collaborate as a team, to understand the business environment and to make and implement business decisions. It has been argued that ERPsim follows the problem-based approach and brings students to learn higher levels of cognitive abilities with respect to enterprise systems (Léger et al., 2012), enhanced supply chain collaboration skills (Caya et al., 2014) and emotional control in ERP decision making (Léger et al., 2014).

The ERPsim simulation for big data seeks to create a dynamic big data environment using SAP software that can simulate the velocity of big data in order to enable the learner to achieve the highest levels of the taxonomy of business analytics. To find its place in this era of big data, SAP put forth a new tool for database architecture, SAP HANA (SAP, USA). SAP HANA is an in-memory database platform that was released in 2010. It allows for real-time analytics of both structured and unstructured data. In-memory analytics refers to a new way through which computers are managing data and applications by keeping data in their main memory instead of regularly having to access the hard drive to retrieve them (vom Brocke et al., 2014). One of the benefits of the HANA technology is that it is no longer necessary to extract the transactional data into a data warehouse in order to perform analytical processes. It is now possible to perform big data analysis directly from the HANA infrastructure which supports both the transactional and analytical processes. New software technologies, such as SAP HANA design studio (creation of analytical views) and SAP LUMIRA (data visualization), enable the extraction and analysis of big transactional data. By combining these new tools with the ERPsim simulator, students can learn how to exploit big data. The ERPsim for big data game is also aligned with Bliemel’s definition of a good educational game (Bliemel and Ali-hassan, 2014) through the following elements: it presents enough of a challenge for most students, allows students control over actions, presents clear goals, shows feedback in the form of financial statements, is a cooperative experience and develops new skills.

2. The pedagogical challenges of teaching business intelligence

There is an extensive literature on end-user training and training methods in the IS literature, and among the various training approaches, enactive learning has been encouraged in recent years (Gupta et al., 2010). Grounded in social cognitive theory (Bandura, 1986), enactive learning builds upon a problem-based approach, in which answers are not to be memorized, but rather in which learners are forced to determine what they need to know in order to find the answers by themselves (Hmelo-Silver et al., 2007). An enactive approach provides the user with the possibility, by trial-and-error, to receive constant feedback in response to their actions, thereby allowing learning by doing through detection and correction of errors (Léger et al., 2014). Optimally, enactive problem-based learning builds on a student-centered approach, where students are in a small group setting and have to collaborate on a very authentic problem (Léger et al., 2012). In the field of information systems, research has shown that enactive training is a very effective approach to acquire skills and competences in using a new business software (Gupta and Bostrom, 2013; Léger et al., 2014).

While the approach itself is gathering support, to our knowledge, most of the available enactive pedagogical material to assist in teaching business intelligence is incomplete. The current pedagogical material available for teaching business analytics does provide various opportunities to put into practice business analytical skills using data sets provided to the students. However, this data have often been pre-generated or adapted for pedagogical purposes and, in most cases, these data sets are static. They represent data available for a company at a specific point in time. Students are tasked to explore and discover opportunities in the data.

Undisputedly, these pedagogical activities can help achieve some pedagogical goals. However, they can fall short in inducing more complex forms of learning and could be improved. The hierarchical complexity of pedagogical objectives is illustrated in the Bloom’s taxonomy of cognitive processes (see Figure 1).

Like the original taxonomy, “the revision is a hierarchy in the sense that the six major categories of the cognitive process dimension are believed to differ in their complexity, with remember being less complex than understand, which is less complex than apply, and so on” (p. 215) (Krathwohl, 2002). Thus, educators or trainers should aim to have their students attain each ability level in order for them to succeed in using this knowledge in their future careers. Different types of teaching need to be put forward in order to guide the students through all phases of learning. It could be argued that class activities requiring the implementation of business intelligence methods on a static database can guide students to reach the first three levels of the taxonomy: “remember,” “understand” and “apply.” For example, they can remember the terminology, menus and procedures specific to the software, understand the functionalities of the system and apply the statistical methods presented in class to generate results. However, in the field of business analytics, the higher level abilities, such as “analyze,” “evaluate” or “create,” are necessary for students to be appropriately trained in management, visualization and interpretation of big data. Well-trained students should be able to analyze the information to correctly identify a problem, to evaluate the problem and identify possible solutions, and to create and develop various ways of representing the data so others can understand it and collectively make better decisions in the long run (Oceans of Data Institute, 2014). These skills will be essential for their ability to handle the complexity of the tasks and real world problems they will encounter as business intelligence professionals (Charland et al., 2015). To obtain these skills while interacting with big data, enactive training approaches are likely to be required.

Big data is commonly defined as the combination of volume (a large quantity of data), variety (multiple types of data) and velocity (the speed at which data are created) (Laney, 2001). With static data sets, students can be provided with volume and variety of data, but not easily with velocity. Velocity is characterized not only by the speed at which data are generated, but also by the dynamic nature of data and how it evolves as a consequence of business decisions. By regularly feeding new data to the students, some degree of velocity may be achieved, but the data would typically be predefined and would not reflect actual decisions made by the students. In order to allow students to attain the highest levels of the taxonomy, the use of unstructured and complex dynamic problems with multiple solutions appears to be an interesting alternative to static data sets, as it links the data to analyze with the decisions they make while resolving the problems.

2.1 ERP technology as platform for enactive learning of BI concepts

In 2012, the Business Intelligence version of the ERPsim game was developed to allow quasi real-time data analytics. It relies on extract, transfer and load (ETL) to provide updated data at the end of each virtual day (each day is about one minute long) and stores the data in a Microsoft SQL database. Using Microsoft Excel, the students are able to extract views from the Microsoft SQL database and refresh these views every minute. This approach offers the ability to develop dashboards that connect to live data from the game, opening a new array of pedagogical possibilities. The Business Intelligence version of the ERPsim game can be used to teach dashboard design principles and business intelligence with a real enterprise system and a live data set.

In this paper, we present ERPsim for big data game as the next step. It enables real-time analytics on raw transactional data. Instead of using an ETL approach, where data used for analysis are available after the ETL process is completed, analysis can be performed directly on transactional data. The information cubes (analytical views in the HANA naming convention) are designed to provide meaningful data to students, data that they are free to explore and visualize as they desire. Visualization tools, such as SAP Lumira, allow this exploration and understanding of the data, and ultimately supports the decision-making process during the simulation.

3. ERPsim for big data: a PD approach

As of 2015, more than 1,000 instructors have been trained to use ERPsim for teaching in more than 220 universities and colleges worldwide. Several of these instructors have been using the original game in their classroom for many years and are often the source of incremental innovation in the simulation.

Early in the phases of development of ERPsim for big data, we took the decision to involve several of these experienced instructors as co-designers in order to ensure that the new simulation game would meet the pedagogical objectives presented in Section 2. These collaborators had extensive experience with ERPsim; they teach ERPsim on a regular basis to a variety of levels and disciplines of study, and have identified key aspects of this knowledge that students need to experience in the game. It was determined that they would be useful in identifying the type of data needed in the cubes for this game as well as the optimal ways to consolidate a large variety of needs into an efficient set of cubes.

As the development of ERPsim for big data got underway, PD was identified as a useful method to involve these co-designers. PD originated in Scandinavia as a tradition to involve the end-user of a technological artifact in its design. It has been defined as a collaboration between the designers and the people who will be using the finished product through an iterative process that often involves brainstorming conversations and prototyping (Simonsen and Robertson, 2013). PD requires reciprocal learning for both the designers and users before stakeholders can engage in democratic and creative sessions (Carmel et al., 1993). All participants thus become co-designers as they have an equally vested interest in the outcome of the process.

For our purpose, two PD events were organized. A design workshop took place in the early phases of development to clearly identify and consolidate the needs of all the co-designers as well as jumpstart the development. Following that, a beta test was put into place along with an accompanying blog to facilitate the conversation between co-designers.

3.1 Participatory co-design workshop

3.1.1 Participants

The PD workshop was held between June 16 and June 19, 2014. In total, 31 participants from 24 different universities and six different countries (USA, Canada, Germany, Switzerland, Indonesia and Finland) participated in the workshop. The workshop was the occasion for long term and experienced users of the ERPsim simulations to learn about new technological improvements and pedagogical innovations in the ERPsim simulations games.

3.1.2 Procedure and methods

Step 1 – development of reciprocal learning by the co-developers

To kick off the workshop, researchers presented the revised bloom taxonomy and the overall objective to develop a serious game material to enable a business student to develop abilities at each level of the taxonomy. To create reciprocal knowledge with regard to the existing ERPsim game, we began the PD session by having the participants play the most recent version of the simulation game, the ERPsim Manufacturing game (Léger et al., 2011). In this game, the teams have to plan production and then purchase the required raw materials in order to get ready-to-eat muesli boxes available for sale. Teams compete for customers (i.e. grocery stores). During the workshop, participants used the Business Intelligence version of ERPsim. As explained earlier, this technology provides analytic visualization based on MS Excel and an SQL Server database. During the simulation game, the ERPsim simulator pushes data into an SQL Server database. This database thus contains the historical data pertaining to the ongoing simulations. A Microsoft Excel worksheet is then used to extract and manipulate views (or queries) already available on the SQL Server database. The views are displayed in the MS Excel file in the form of pivot tables. Using the pivot tables, the participants can create multiple graphs and tables, use conditional formatting to create alerts, etc. The use of MS Excel offers multiple benefits, as it is an easily accessible programming tool. However, the data download is slow and not designed with the intent to provide easy insight to act upon. Furthermore, data are only available after some delay (two to three minutes). The objective of initially running this game with the MS Excel technology in our workshop was to make sure that all participants were familiar with the latest version of the technology, shared a similar understanding of the concept of cubes, and of the benefits and limits of using existing analytical technology in ERPsim games.

Step 2 – ideation process

After this reciprocal learning activity, we moved on to the ideation stage of the process. The moderator, one of the creators of the game, first provided an overview of the ideation process in which the group would participate. The moderator then asked each participant to participate in a brainstorm activity regarding the information architecture required for a new version. Specifically, the participants had to identify useful sets of measures and dimensions. The information cubes around which the business analytics are built refer to two main concepts: measures (quantity sold, sales revenues, etc.) and dimensions (time, geographic, type of products, etc.). So a typical combination of measure (bold) and dimensions (italic) is something like “quantity sold by product over time” “quantity in stock by product and area” or “number of different products available on the market at any given time.” All in all, about 40 distinct combinations were proposed by the participants.

Step 3 – consolidation of requirements

Building on requirements extraction approaches proposed by Hartson and Pyla (2012), the group then worked on consolidating the informational requirements. Specifically, the group of co-designer filtered, sorted and regrouped the proposed measure and dimension combinations. Each participant was asked to identify which combinations among the proposed list were the most important and useful. Combinations were also sorted by category. Some combinations could be grouped into one theme, cube or view. In the end, we collectively came up with seven different cubes, cubes that together contained a large number of the proposed measure and dimension combinations. The cubes were drawn on the board by the moderator (see Plate 1).

Step 4 – review existing information architecture

When developing information architecture, it is crucial to reuse existing structure developed by the software manufacturer. The main goal is to leverage existing design and minimize future maintenance of the architecture. Thus, the next step was to examine the existing cubes already available in SAP. Using Lumira, a data-visualization tool that connects to ERPsim for big data to retrieve real-time data from the SAP ERP system, users can create visualizations using a drag-and-drop interface. While the participants were seated in the conference room, one of the lead researchers on the ERPsim for big data project, acting as moderator, and the ERPsim lab operations manager (OM) were at the front of the room. The OM was using Lumira in order to visualize data from a live game, data that were provided by some of the existing cubes in SAP. The objective of the exercise was to come up with specific data visualizations within Lumira, and participants were invited to propose ideas and debate on how to reuse or extend existing informational architecture in SAP HANA. The role of the moderator was to lead the conversation and provide guidance. Once a consensus was reached, the OM implemented the suggested presentation live, leading to more comments from the audience and adjustments by the OM. There was no structured layout for the workshop so as to allow the conversation to flow, the moderator and OM improvised when to move on to a new topic and which topic to move on to.

Step 5 – implementation of co-design solution

The final step was to create seven new cubes. The conceptual definition of the seven cubes was drawn on the classroom board (see Plate 1). The proposed designs were used to create a fast prototype version in SAP HANA to ascertain the feasibility of the proposed cubes. The implementation was performed live by an analyst so that participant could learn and comment on the detailed configuration of the solution. Modeling in SAP HANA is essentially done by creating “attribute views,” “analytical views” and “calculation views,” which are referred to as cubes. Using the tables from the database as their source, the cubes are created using a business use case to model business logic and provide meaningful reports (SAP SE, 2014). When implementation was complete, participants were able to access the cubes and provide immediate feedback on the configuration. Immediate adjustments were made to increase the quality of the solution.

3.1.3 Results and discussion

The main benefits of the workshop format stemmed from the face-to-face group setting. This setting allowed not only a more pedagogically optimal version of the game to be developed, but also gave the developers ready answers if changes discussed in the workshop are ever requested by others. In a more traditional setting where software testers simply send comments to a developer, this type of reflection would not have taken place. For example, based on a suggestion during the workshop, an existing view that used to provide information on the market every five days was reconstructed to allow the report to be produced every day, making it possible to have a more granular analysis. However, an experienced user and collaborator commented that having that level of details will often cause students to simply follow the market without thinking and anticipating the market’s reactions. This, in turn, would lead to students not learning as much about the decision-making process. In the end, the consensus was to go back to the less granular, but more educational, five day report.

This type of group reflection leads to a large number of innovation iterations in a short period of time and makes the development much more efficient. Another advantage of a face-to-face workshop of this type, as opposed to online discussions where a very vocal minority can seem like they represent the majority, is that the moderator can encourage everyone to voice their opinion. Finally, participants are fully immersed and have no other distraction, as opposed to an at-home beta test where other obligations can limit availability for discussion. Unfortunately, this process might be difficult to reproduce for a new game that does not already have a significant number of users who are willing to participate in the design of a new iteration. It also requires participants to have a common ground in order to ensure they all “speak the same language.”

In total, eight data cubes were developed based on the results of the co-design workshop. The next stage was to have these cubes tested in live setting. The next section describes the second phase of the co-development process.

3.2 Beta test of co-design solution

In the fall of 2014, a beta test of ERPsim for big data was conducted with a small sample of experienced instructors. These instructors used the new game in their classrooms for the semester as they saw fit. The goal was for them to evaluate the newly developed solution and to collaborate with the design team to further optimize these informational cubes. Also, we aimed at assessing if ERPsim for big data would meet the pedagogical objectives targeted by this new simulation, i.e. the analytical abilities at the different stages of the taxonomy, and more specifically to what extent this new version better meets these objectives as compared to previous ERPsim games.

3.2.1 Participants

The first phase of the beta test included 11 instructors from different universities throughout Canada and the USA. All of the instructors were university professors and have accepted to provide feedback from their experience by answering an online questionnaire and by participating in a co-design blog. The participating instructors were teaching to classes ranging from 15 to 100 students and spanning all levels of undergraduate and masters classes. Six classes were in IT/IS departments, one in management and four in other. The instructors had between 7 and 25 years of experience as teachers and all have been using SAP and ERPsim for over two years.

3.2.2 Procedure and method

To accompany the beta test, a collaborative blog was put in place. Each user of the beta test was granted access so they could create entries and post comments on other entries. No restrictions on content were imposed.

To document the students’ learning, a web questionnaire was set up on Unipark (Questback, Germany), so it could be sent by e-mail and answered online. The first portion documented the makeup of the class (number of students, level, duration of use in class). The second portion of the questionnaire was the Self-assessment scale of learning in the context of an ERP business simulation game reproduced from Cronan et al. (2012) which requires participants to rate, using a one to seven Likert scale, the abilities they believe students developed while using the traditional ERPsim game vs while using ERPsim for big data. That instrument has three sections: the enterprise systems management knowledge, the business process knowledge and the SAP transaction skills. A cumulative score was calculated for each section for each game. The third portion required participants to rate their agreement with statements on the ease of use of ERPsim for big data and the fourth portion asked for comments on that use. Finally, the fifth portion documented the professor’s experience teaching and using SAP and ERPsim. A paired Wilcoxon test was used to evaluate the comparison between the traditional ERPsim game and the new ERPsim for big data measures. We also used non-parametric correlations to evaluate the impact of the professor’s experience, the makeup of the class and the professor’s participation in the blog onto the measures.

3.2.3 Results and discussion

When comparing the learning that occurs with a traditional ERPsim game and with the new ERPsim for big data game, a significant difference can be seen for the enterprise systems knowledge (p=0.02) with perceived student learning increased for the ERPsim for big data game. For business process knowledge (p=0.22) and the SAP transaction skills (p=0.15), we cannot conclude that there is a significant difference in learning. These results show that this new version of the game is an improvement on the traditional version and that the increase in enterprise system knowledge is not counterbalanced by a drop in another area of knowledge. Interestingly, the newly designed information architecture appears to help the learner get a broader understanding of the purpose of the enterprise system. The perception might be associated with the fact that students had a better sense of the insight to action when using the new real-time visualization software when making decision in the enterprise system.

For the teachers’ perceptions, four correlations were found to be significant. First, the number of years of experience with ERPsim was negatively correlated with the agreement with the statement “I found ERPsim for big data useful as a pedagogical tool” (r=−0.83, p=0.003). Second, the number of blog posts by a professor was negatively correlated with most scores of the learning assessment for both games (BP_ERPsim: r=−0.65, p=0.03; TS_ERPsim: r=−0.59, p= 0.05; ES_BigData: r=−0.61, p= 0.06; BP_BigData: r=−0.73, p=0.02; TS_BigData: r=−0.77; p= 0.01). Both of these could be explained by professors who are participating in the co-design process or who are more experienced with the game being more critical of their students’ learning as they have a more thorough understanding of the optimal learning that can be derived from these simulations. Third, the number of blog posts was also positively correlated with the agreement with the statement “I found the use of ERPsim for big data an improvement on my previous classes” (r= 0.76, p=0.01), this would suggest that involvement in the PD process influences their perception of the added value of the new game.

The following summarizes the comments made by the professors to the qualitative questions we asked. To the question “ERPsim for Big data allows students to work with real-time analytical tools, as a user of the ERPsim game, what do you believe this new tool brings to your students’ skills?,” most participants pointed out the crucial role of real-time data in decision making. Almost all respondent answered that ERPsim for big data helped students acquire the ability to leverage real-time data in order to create instant analysis and make better decisions, “this tool allowed students to better understand the simulation and recognize the value of analytics.” Some participants mentioned that this tool brought the development of “visualization skills” to their students using a “simple drag and drop technology.” Finally, one participant mentioned its students gained the “ability to share a common perception on the company” and the “ability to rapidly iterate in their collaborative problem solving.”

To the question: “What do you believe is the added value of real-time analytics to your classroom?,” a number of respondents referred to the experiential aspect of the class. “The real-time analytics is an essential part of the classroom. It highlights one of the key value of information system.” “Students are able to experience real time analytics in a dynamics context,” “[…] without this the students might think ERP has no ability to provide real-time analytics.” One participant added: “it gives my students familiarity with a new business tool so that they can speak confidently in a job interview.”

To the same question, some users mentioned the added value of the ease of use of the game. Finally, the central themes of immediacy (“the importance of utilizing high velocity data,” “Immediate feedback from business decisions”) and of visualization (“ability to visualize business in motion and see patterns”) are mentioned similarly to the previous question.

To the question “How did the use of ERPsim for Big data change the enthusiasm/interest/motivation of your students? Do you think it affected the mood and attitudes of the class when compared to your previous use of ERPsim game?,” most participants observed an improvement in attitude. “They felt somewhat more under control” was the response of one participant. “Students were very excited to be able to see how their decisions were affecting the performance of their organization” was the answer of another. The ease of use and the flexibility of the reporting enabled the students to see the impact of their decision. “They are not trapped in the limitations of the SAP GUI and can adapt their own decision support system. Hence they are more engaged.” Two respondents, however, remained unsure about the impact of the change on their students’ perceptions.

To the question “How would you use it differently next time?,” the responses from participants vary considerably. One wishes to highlight the technology behind HANA. One plans no changes next time. One wants to explore predictive tools to run regression and statistical analysis. One “would like the students have the ability to develop cubes and then develop their own analytical approach.” Finally, one “intends to demonstrate to students that while SAP and HANA facilitates timelier and more accurate business decisions within the business enterprise, it will still be up to them, as managers and supervisors to communicate with each other.”

Finally, we sought comments on future iterations of the game design, here are some comments from the beta test participants that identify a commonly requested feature that will be discussed by the design team. We asked “What changes would you like to see in ERPsim for Big data?” While some are uncertain about the changes they would like to see in ERPsim for big data. Others have very precise requests such as “put the news and raw material prices so they can be piped in through Lumira dashboards” or “create location data for customer so we can geomap sales.” Two participants proposed to use alternative visualization tools such as Tableau (Seattle, WA, USA): “Tableau would increase the appeal of using ERPsim in a business intelligence class that already uses Tableau as the primary visualization tool.” Three respondents would like to see more material developed in order to make “ERPsim [for Big data] as the cornerstone of an applied Business Analytics class.” In particular, the “ability to integrate predictive analysis features” and “ability for students to construct their own views and dig more into the development side of HANA” were proposed.

In short, while it is not possible to conclude that the new version completely addresses all the learning stages of the taxonomy, this new version of the game appears to provide students with more enterprise system knowledge and get a better appreciation for the role of real-time data in business decision making. Comments by participants identified a number of additional features that would be appreciated by users, in particular, the ability to create additional views and/or connect to alternative visualization tools. This shows that the beta test and the accompanying survey were beneficial for the design team, while the improvement in learning confirms that the game is reaching its goals and paves the way for the next steps in development. Following the beta test, the eight cubes were optimized in light of the collaborators comments.

4. Conclusions

By employing a PD workshop and a closely monitored beta test, we gained a better understanding of the needs of the end users as well as benefited from their knowledge and experience in complementary fields of expertise. This allowed the development of this new game to be much faster than otherwise envisioned. It also set the tone for a much more active adoption of the technology, with users quickly informing the designers of any bugs. Finally, by having all the co-designers in the same room at the workshop, their needs were more easily reconciled, leading to less redundant cubes and a more efficient development.

Following the 2014 summer workshop, the final comments from the discussion were integrated in the development and optimization of the cubes. Specifically, they oriented the selection of the business cubes or views that will be included in the general release. It also provided the development team with enough comments to confirm or invalidate the needs for information in some of the views.

In preparation for the general release, all of the cubes initially developed have been reengineered to match the requirements defined by the team and tested with the co-designers. Additionally, cubes that were ideated during the workshop have been provided as an initial release during the beta period, and are now being optimized. Improvements to the simulator are also needed to support the new requirements; this is being done in parallel of the cube development. September 2015 marked the release of ERPsim for big data. The final version is described in Michon et al. (2015).

We conclude with a number of recommendations for future serious game developers who wish to employ similar activities to optimize their development. In general, involving teachers and professors, rather than students, as co-designers allowed us to circumvent some of the communication difficulties identified for PD in serious game design, as “serious game designers must be fluent with both domain content and game design,” while end users “may lack one or both of these forms of knowledge” (Khaled and Vasalou, 2014). The downside, however, of such a process is that expectations were much higher and workshop participants hoped their desired applications would be included in the game.

4.1 Recommendations for a PD workshop activity

• Someone who is part of the design team and who has a very good understanding of what is currently available should do the moderation to lead the conversation in the right direction.

• Concerns about the implementation of the suggestions should not be discussed during the workshop to allow creative out-of-the-box thinking. As mentioned by the OM: “say yes to everything, discuss feasibility later.” However, there should be an initial disclaimer that not all ideas will be kept to avoid disappointed expectations later on.

• According to the OM, it is important to “have an open mind and humility about your product, since the participants will challenge design decisions at some point. By providing them with an environment to freely express their opinions about the product, you allow yourself to get the most out of the activity, and discussions may explore possibilities that had not been considered initially.”

• To avoid concentrating on note taking, record the discussion.

• The more users are comfortable with the functional technology, the better they will be able to comment on the content. Participants should be given ample time to familiarize themselves with a prototype ahead of time.

• Having participants play a preliminary version of the game before the brainstorming session allowed them to really understand the possibilities of such a tool.

4.2 Recommendations for a closely monitored beta test

• A user-friendly way to report bugs in the system is very important; however, it might be preferable to have that separately from a comment section on possible design improvements.

• An objective empirical evaluation of the game should be planned out in advance and can be a great tool to demonstrate the pedagogical advantage such a game provides.

• Qualitative comments, however, are useful in identifying specific elements of the game that require improvement.

The development of this game was done with enactive learning in mind. The next step is to evaluate students’ learning and to perform a direct comparison with vicarious learning alone for the same topic to adequately identify which step of the Bloom Learning Taxonomy (Krathwohl, 2002) is reached by students using this game as well as whether or not students do learn more through enactive learning when applied to business intelligence and big data management. For example, with the suggestion of allowing students to create their own views that was brought up in the beta testing phase, this was implemented (Michon et al., 2015) and we believe by having them be creative they will attain the highest understanding of this topic.

This paper contributes to research and practice in the field of educational games development. PD with experienced instructors as co-designers has proved most helpful in the case of ERPsim for big data and the lessons learned from this development should be taken into consideration by others. It will help them design their games faster, help insure that the game is aligned with their pedagogical needs and help increase adoption once the game is released and using a structured PD method will help consolidate the needs of multiple co-designers without getting overwhelmed. It also contributes to research and practice on the teaching of big data, showing the importance of enactive learning and the importance of an authentic and dynamic business environment. With appropriate pedagogical tools that provide students with optimal skills and attitudes toward business intelligence, we can ensure a highly skilled future workforce.

Figures

Figure 1

Revised Bloom learning taxonomy

Plate 1

Example of a cube drawn at the workshop