Defining Preliminary Research for Digital Game-Based Learning Evaluation: Best Practices
The surge of learners being immersed in computer game contexts for learning has instigated dialogue about the contextually appropriate collection of r.
- Pub. date: November 15, 2019
- Pages: 623-635
- 770 Downloads
- 1033 Views
- 9 Citations
The surge of learners being immersed in computer game contexts for learning has instigated dialogue about the contextually appropriate collection of reliable and valid data to inform education-based decisions. The purpose of this article is to develop educational practitioners’ understanding of preliminary research work, and to inform educational researchers about design and reporting of preliminary research work, in the context of reported preliminary studies on Digital Game-Based Learning Interventions (DGBLIs). First a checklist of processes for the reporting of preliminary studies is provided. Second, a summary is offered of the characteristics of each type of preliminary study including the description, objectives, and methodology. Third, an example from peer-reviewed literature is identified of each type of preliminary study relevant to DGBLIs and conducted within the past five years. Evident from the examples selected, educational researchers and practitioners are best advised to recognize the characteristics of preliminary studies — pilot work, feasibility study, pilot study, pilot trial, and field test — to better inform DGBLIs before embarking on a full-scale study, and to meet the need of educational practitioners for concrete evidence about DGBLIs.
Keywords: Preliminary research studies, digital game-based learning interventions, pilot study, field test, education methodology
References
Alber, J., Watson, A., Barnett, T., Mercado, R., & Bernhardt, J. (2015). Development of a coding instrument to assess the quality and content of anti-tobacco video games. Cyberpsychology, Behavior, and Social Networking, 18(7), 417-425. https://doi.org/10.1089/cyber.2015.0051
Arain, M., Campbell, M., Cooper, C., & Lancaster, G. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology, 10(1), 67-74. https://doi.org/10.1186/1471-2288-10-67
Arnold, D., Burns, K., Adhikari, N., Kho, M., Meade, M., & Cooke, D. (2009). The design and interpretation of pilot trials in clinical research in critical care. Critical Care Medicine, 37(1 Suppl), 69-74. https://doi.org/10.1097/CCM.0b013e3181920e33
Astor, P., Adam, M., Jercic, P., Schaaf, K., & Weinhardt, C. (2013). Integrating biosignals into information systems: A NeuroIS tool for improving emotion regulation. Journal of Management Information Systems, 30(3), 247-277. https://doi.org/10.2753/MIS0742-1222300309
Barney, J. (2018). Editor’s comments: Theory contributions and the AMR review process. Academy of Management Review, 43(1), 1-4. https://doi.org/10.5465/amr.2017.0540
Bertoni, M., Bordegoni, M., Johansson, C., & Larsson, T. (2008). Pilot specifications definition guidelines for the implementation of a KEE solution in the aeronautical domain. In: Fred J. A. M. van Houten (Ed.), CIRP Design Conference (pp.1-7). Enschede, The Netherlands: University of Twente.
Bingham, P., Lahiri, T., & Ashikaga, T. (2012). Pilot trial of spirometer games for airway clearance practice in cystic fibrosis. Respiratory Care, 57(8), 1278-1284. https://doi.org/10.4187/respcare.01263
Blekken, L., Nakrem, S., Gjeilo, K., Norton, C., Moekved, S., & Vinsnes, G. (2015). Feasibility, acceptability, and adherence of two educational programs for care staff concerning nursing home patients’ fecal incontinence: a pilot study preceding a cluster-randomized controlled trial. Implementation Science, 10(1), 1-13. https://doi.org/10.1186/s13012-015-0263-8
Boendermaker, W., Gladwin, T., Peeters, M., Prins, P., & Wiers, R. (2018). Training working memory in adolescents using serious game elements: Pilot randomized controlled trial. JMIR Serious Games, 6(2), e10. https://doi.org/10.2196/games.8364
Bugge, C., Williams, B., Hagen, S., Logan, J., Glazener, C, Pringel, S., & Sinclair, L. (2013). A process for decision-making after pilot and feasibility trials (ADePT): Development following a feasibility study of a complex intervention for pelvic organ prolapse. Trials, 14, 1-29. https://doi.org/10.1186/1745-6215-14-353
Conn, V., Algase, D., Rawl, S., Zerwic, J., & Wyman, J. (2010). Publishing pilot intervention work. Western Journal of Nursing Research, 32(8), 994-1010. https://doi.org/10.1177/0193945910367229
Connolly, T., Boyle, E., MacArthur, E., Hainey, T., & Boyle, J. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. https://doi.org/10.1016/j.compedu.2012.03.004
Cowley, B., Fantato, M., Jennett, C., Ruskov, M., & Ravaja, N. (2014). Learning when serious: Psychophysiological Evaluation of a Technology-Enhanced Learning Game. Educational Technology & Society, 17(1), 3-16.
Doody, O., & Doody, C. (2015). Conducting a pilot study: case study of a novice researcher. British Journal of Nursing, 24(21), 1074-1078. https://doi.org/10.12968/bjon.2015.24.21.1074
Duemer, L., & Zebedi, A. (2009). The pragmatic paradigm: An epistemological framework for mixed methods research. Journal of Philosophy and History of Education, 59, 163-168.
Eldridge, S., Lancaster, G., Campbell, M., Thabane, L, Hopewell, S., Coleman, & Bond, C. (2016). Defining feasibility and pilot studies in preparation for randomised controlled trials: Development of a conceptual framework. PLOS ONE, 11(3), 1-22. https://doi.org/10.1371/journal.pone.0150205
Franzen, A., & Pointner, S. (2013). The external validity of giving in the dictator game: A field experiment using the misdirected letter technique. Experimental Economics, 16(2), 155-169. https://doi.org/10.1007/s10683-012-9337-5
Golding, K., Kneebone, I., & Fife-Schaw, C. (2016). Self-help relaxation for post-stroke anxiety: a randomised, controlled pilot study. Clinical Rehabilitation, 30(2), 174-180. https://doi.org/10.1177/0269215515575746
Guest, G. (2012). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7(2), 141-151. https://doi.org/10.11771558689812461179
Janssen, A., Shaw, T., Goodyear, P., Kerfoot, B. P., & Bryce, D. (2015). A little healthy competition: using mixed methods to pilot a team-based digital game for boosting medical student engagement with anatomy and histology content. BMC Medical Education, 15(1), 1-10. https://doi.org/10.1186/s12909-015-0455-6
Jasem, Z., & Delport, S. (2019). Mother's perspectives on the play of their children with Attention Deficit Hyperactivity Disorder. Occupational Therapy International, 2019, 1-10. https://doi.org/10.1155/2019/6950605
Katajapuu, N., Granholm, P., Hiramatsu, M., Ishihara, E., Hirayama, J., Pitkakangas, P. … Luimula, M. (2016). Brain trainer exercise game. Field tests in Finland and Japan. International Journal of Chemistry and Chemical Engineering Systems, 1, 39-45.
Kerr, D. (2013). Identifying key features of students’ performance in educational video games and simulation through cluster analysis. Journal of Educational Data Mining, 4(1), 144-182.
Kim, Y. (2011). The pilot study in qualitative inquiry: identifying issues and learning lessons for culturally competent research. Qualitative Social Work, 10(2), 190-206. https://doi.org/10.1177/1473325010362001
Lee, E., Whitehead, A., Jacques, R., & Julious, S. (2014). The statistical interpretation of pilot trial: should significance thresholds be reconsidered? BMC Medical Research Methodology, 14(1), 41-49. https://doi.org/10.1186/1471-2288-14-41
Main, S., O’Rourke, J., Morris, J., & Dunjey, H. (2016). Focus on the journey, not the destination: Digital games and students with disability. Issues in Educational Research, 26(2), 315-331.
Montgomery, C., & Smith, L. (2015). Bridging the gap between research and practitioners. Die Unterrichtspraxis/Teaching German, 48(1), 100-113. https://doi.org/10.1111/tger.10183
Paulic, D. (2017). Pilot trial of a knowledge sharing intervention program with digital games. (Doctoral dissertation, University of Phoenix). Retrieved from https://eric.ed.gov/?id=ED586859
Petri, G., & Gress von Wangenheim, C. (2016). How to evaluate educational games: A systematic literature review. Journal of Universal Computer Science, 22(7): 992-1021.
Petri, G., & Gress von Wangenheim, C. (2017). How games for computing education are evaluated? A systematic literature review. Computers & Education, 107, 68-90. https://doi.org/10.1016/j.compedu.2017.01.004
Pritchard, K., & Whiting, R. (2012). Autopilot? A reflexive review of the piloting process in qualitative e-research. Qualitative Research in Organizations and Management, 7(3), 338-353. https://doi.org/10.1108/17465641211279798
Procci, K., Lakhmani, S., Hussain, T., & Bowers, C. (2014). Opening cinematics: Their cost-effectiveness in serious games. Simulation & Gaming, 45(1), 93-124. https://doi.org/10.1177/1046878113508515
Rajasekar, A., & Kumar, V. (2019). Randomized controlled trials: Gold standard of evidence. Drug Invention Today, 12(6), 1215-1217.
Randel, B., & Clark, T. (2013). Measuring classroom assessment practices. In J.H. McMillan SAGE handbook of research on classroom assessment (pp. 145-163). Thousand Oaks, CA: Sage Publications, Inc. https://doi.org/10.4135/9781452218649.n9
Read, J., & Shortell, S. (2011). Interactive games to promote behavior change in prevention and treatment. Journal of the American Medical Association, 305(16), 1704-1705. https://doi.org/10.1001/jama.2011.408
Reeves, B., Cummings, J., Scarborough, J., & Yeykelis, L. (2015). Increasing energy efficiency with entertainment media: An experimental and field test of the influence of a social game on performance of energy behaviors. Environment and Behavior, 47(1), 102-115. https://doi.org/10.1177/0013916513506442
Richard V., & Belanger, M. (2018). Accepting research: Teachers’ representations of participation in educational research projects. International Journal of Educational Methodology, 4(2), 61-73. https://doi.org/10.12973/ijem.4.2.61
Schneider, S., Schmidli, H., & Friede, T. (2013). Blinded and unblinded internal pilot study designs for clinical trials with count data. Biometrical Journal, 55(4), 617-633. https://doi.org/10.1002/bimj.201200189
Serret, S., Hun, S., Lakimova, G., Lozada, J., Anastassova, M., Santos, A. … Askenazy, F. (2014). Facing the challenge of teaching emotions to individuals with low- and high-functioning autism using a new serious game: a pilot study. Molecular Autism, 5(1), 1-35. https://doi.org/10.1186/2040-2392-5-37
Sims, W., Riley-Tillman, C., & Cohen, D. (2017). Formative Assessment using direct behavior ratings: Evaluating intervention effects of daily behavior report cards. Assessment for Effective Intervention, 43(1), 6-20. https://doi.org/101177/1534508417708183
Smith, P., Morrow, R., & Ross, D. (Ed.). (2015). Preliminary studies and pilot testing. In field trials of health interventions: A toolbox (3rd ed.). Oxford, UK: OUP Oxford.
Stouck, J. (2019). Complementary learning: Piloting a blended format for Canadian composition courses. Canadian Journal for the Scholarship of Teaching & Learning, 10(1), 5-20. https://doi.org/10.5206/cjsotl-rcacea.2019.1.7992
Strader, T. (2011). Digital product management, technology, and practice: Interdisciplinary perspectives. New York, NY: Business Science Reference. https://doi.org/10.4018/978-1-61692-877-3
Sullivan, G., & Sargeant, J. (2011). Qualities of qualitative research: Part I. Journal of Graduate Medical Education, 3(4), 449-452. https://doi.org/10.4300/JGME-D-11-00221.1
Toomey, M. (2017). Engaging the enemy: Computer games in the English classroom. Literacy learning: The Middle Years, 25(3), 38-49.
Tuzun, H., Barab, S., & Thomas, M. (2019). Reconsidering the motivation of learners in educational computer game contexts. Turkish Journal of Education, 8(2), 129-159. https://doi.org/10.19128/turje.546283
Vate-U-Lan, P. (2015). Transforming classrooms through game-based learning: A feasibility study in a developing country. International Journal of Game-Based Learning, 5(1), 46-57. https://doi.org/10.4018/ijgbl.2015010104.
Whitehead, A., Sully, B., & Campbell, M. (2014). Pilot and feasibility studies: Is there a difference from each other and from a randomized controlled trial? Contemporary Clinical Trials, 38(1), 130-133. https://doi.org/10.1016/j.cct.2014.04.001
Wiens, P. (2014). Using a participant pool to gather data in a teacher education program: The course of one’s school’s efforts. Issues in Teacher Education, 23(1), 177-206.
Wiseman, A., Whitty, G., Tobin, J., & Tsui, A. (2010). The Uses of Evidence for Educational Policymaking: Global Contexts and International Trends. Review of Research in Education, 34(1), 1-24. https://doi.org/10.3102/0091732X09350472
Wolfe, B. (2013). The value of pilot studies in clinical research: a clinical translation of the research article titled “In search of an adult attachment stress provocation to measure effect on the oxytocin system.” Journal of the American Psychiatric Nurses Association, 19(4), 192-194. https://doi.org/101177/1078390313493248
Wood, C., Mentzelopolous, M., & Protopsaltis, A. (2015). EdCCDroid: An education pilot prototype for introducing Code-Combat using LUA. In D. Preuveneers (Ed.), Workshop Proceedings of the 11th International Conference on Intelligent Environments (pp. 353-340). Prague, Czech Republic: University of Westminster.