Development and Validation of Instruments for Assessing the Impact of Artificial Intelligence on Students in Higher Education
The role of artificial intelligence (AI) in education remains incompletely understood, demanding further evaluation and the creation of robust assessm.
- Pub. date: May 15, 2024
- Online Pub. date: February 06, 2024
- Pages: 197-211
- 706 Downloads
- 2979 Views
- 0 Citations
The role of artificial intelligence (AI) in education remains incompletely understood, demanding further evaluation and the creation of robust assessment tools. Despite previous attempts to measure AI's impact in education, existing studies have limitations. This research aimed to develop and validate an assessment instrument for gauging AI effects in higher education. Employing various analytical methods, including Exploratory Factor Analysis, Confirmatory Factor Analysis, and Rasch Analysis, the initial 70-item instrument covered seven constructs. Administered to 635 students at Nueva Ecija University of Science and Technology – Gabaldon campus, content validity was assessed using the Lawshe method. After eliminating 19 items through EFA and CFA, Rasch analysis confirmed the construct validity and led to the removal of three more items. The final 48-item instrument, categorized into learning experiences, academic performance, career guidance, motivation, self-reliance, social interactions, and AI dependency, emerged as a valid and reliable tool for assessing AI's impact on higher education, especially among college students.
Keywords: Artificial intelligence, item measurement, reliability test, validity test.
0
References
Akour, M. M. (2022). Rasch rating scale analysis of the survey of attitudes toward statistics. EURASIA Journal of Mathematics, Science and Technology Education, 18(12), Article em2190. https://doi.org/10.29333/ejmste/12646
Albano, T. (2020). Introduction to educational and psychological measurement using R. Thetaminusb. http://tinyurl.com/2xcethux
Awang, Z. (2012). Structural equation modelling using AMOS graphic. UiTM Press.
Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79-86. https://doi.org/10.1177/07481756135138
Baghaei, P. (2008). The Rasch model as a construct validation tool. Rasch Measurement Transactions, 22(1), 1145-1146. http://tinyurl.com/479sxke5
Bhandari, P. (2022). What is face validity? | guide, definition & examples. Scribbr. https://tinyurl.com/ye23n5c6
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Lawrence Erlbaum Associates.
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.
Booc, N. B., Sobremisana, K., Ybañez, A., Tolosa, R., Ladroma, S. M., & Caparoso, K. M. (2023). Artificial intelligence-powered calculator application usage in mathematics summative assessments. Iconic Research and Engineering Journals, 6(10), 446-474. https://bit.ly/3unOdDQ
Cliff, N. (1988). The eigenvalues-greater-than-one rule and the reliability of components. Psychological Bulletin, 103(2), 276. https://doi.org/10.1037/0033-2909.103.2.276
Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis. Erlbaum.
Denecke, K., Abd-Alrazaq, A., & Househ, M. (2021). Artificial intelligence for chatbots in mental health: Opportunities and challenges. In M. Househ, E. Borycki & A. Kushniruk (Eds.), Multiple perspectives on artificial intelligence in healthcare: Opportunities and challenges (pp. 115-128). Springer. https://doi.org/10.1007/978-3-030-67303-1_10
Eltahir, M. E., Alsalhi, N. R., Al-Qatawneh, S., AlQudah, H. A., & Jaradat, M. (2021). The impact of game-based learning (GBL) on students’ motivation, engagement and academic performance on an Arabic language grammar course in higher education. Education and Information Technologies, 26, 3251-3278. https://doi.org/10.1007/s10639-020-10396-w
Field, A. (2013). Discovering statistics using IBM SPSS statistics. Sage.
Frost, J. (2023). Cronbach’s alpha: Definition, calculations & example. Statistics by Jim. http://tinyurl.com/3n55dr77
Gaskination's StatWiki. (n.d.). CFA. https://bit.ly/3Oiw54X
George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Allyn & Bacon.
Gerber, N. L., & Price, J. K. (2018). Measures of function and health-related quality of life. In J. Gallin, F. Ognibene & L. Johnson (Eds.), Principles and practice of clinical research (pp. 303-315). Academic Press. https://doi.org/10.1016/B978-0-12-849905-4.00021-6
Goodwyn, F. (2012). Question number two: how many factors? (ED529100). ERIC. https://eric.ed.gov/?id=ED529100
Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis (5th ed.). Prentice-Hall
Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis. Pearson University Press.
Hassan, A. (2011). Kesahan dan kebolehpercayaan item penilaian pembimbing dalam pembelajaran berasaskan kerja (PBK) menggunakan model pengukuran Rasch [Validity and trustworthiness of supervisor assessment items in work-based learning (PBK) using the Rasch measurement model]. USM, Psychometrics Centre, MIMOS & Malaysian Examination Syndicate, MOE.
Hsu, T. C., & Chen, M. S. (2022). The engagement of students when learning to use a personal audio classifier to control robot cars in a computational thinking board game. Research and Practice in Technology Enhanced Learning, 17, Article 27. https://doi.org/10.1186/s41039-022-00202-1
Hwang, G. J., Sung, H. Y., Chang, S. C., & Huang, X. C. (2020). A fuzzy expert system-based adaptive learning approach to improving students’ learning performances by considering affective and cognitive factors. Computers and Education: Artificial Intelligence, 1, Article 100003. https://doi.org/10.1016/j.caeai.2020.100003
Irons, A., & Elkington, S. (2007). Enhancing learning through formative assessment and feedback. Routledge. https://doi.org/10.4324/9780203934333
Jindal, A., & Bansal, M. (2020). Knowledge and education about artificial intelligence among medical students from teaching institutions of India: A brief survey [version 1]. MedEdPublish, 9, Article 200. https://doi.org/10.15694/mep.2020.000200.1
Kairu, C. (2020). Students’ attitude towards the use of artificial intelligence and machine learning to measure classroom engagement activities. In Proceedings of EdMedia+ Innovate Learning (pp. 793-802). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/p/217382.
Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36. https://doi.org/10.1007/BF02291575
Kaiser, H. F. (1960). Varimax solution for primary mental abilities. Psychometrika, 25(2), 153-158. https://doi.org/10.1007/BF02288578
Kang, M., & Im, T. (2013). Factors of learner–instructor interaction which predict perceived learning outcomes in online learning environment. Journal of Computer Assisted Learning, 29(3), 292-301. https://doi.org/10.1111/jcal.12005
Kline, R. B. (2023). Principles and practice of structural equation modeling. Guilford Publications.
Knekta, E., Runyon, C., & Eddy, S. (2019). One size doesn’t fit all: Using factor analysis to gather validity evidence when using surveys in your research. CBE—Life Sciences Education, 18(1), 1-17. https://doi.org/10.1187/cbe.18-04-0064
Kooli, C. (2023). Chatbots in education and research: A critical examination of ethical implications and solutions. Sustainability, 15(7), 5614. https://doi.org/10.3390/su15075614
Lawshe, C. H. (1975). A quantitative approach to con-tent validity. Personnel Psychology, 28, 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
Lennartz, S., Dratsch, T., Zopfs, D., Persigehl, T., Maintz, D., Große Hokamp, N., & Pinto dos Santos, D. (2021). Use and control of artificial intelligence in patients across the medical workflow: single-center questionnaire study of patient perspectives. Journal of Medical Internet Research, 23(2), Article 24221. https://doi.org/10.2196/24221
Linacre, J. M. (2007). A user’s guide to WINSTEPS Rasch Model computer programs. MESA Press
Linacre, J. M. (2012). A user's guide to WINSTEPS® MINISTEP Rasch-model computer programs. Winsteps.Com. https://www.winsteps.com/winman/copyright.htm
Ma, Y., & Siau, K. L. (2018). Artificial intelligence impacts on higher education. In Proceedings of the Thirteenth Midwest Association for Information Systems Conference (pp. 1-5). Association for Information Systems Electronic Library (AISeL). https://aisel.aisnet.org/mwais2018/42
Malhotra, A. (2020, November 3). The promise of artificial intelligence (AI) in education. Linkedin. http://tinyurl.com/22ys5wab
Masero, R. (2023). Evolving Education: The Impact of AI and VR technology on the future of learning. Learning Industry. http://tinyurl.com/4s9uj9nx
Maskey, R., Fei, J., & Nguyen, H. O. (2018). Use of exploratory factor analysis in maritime research. The Asian Journal of Shipping and Logistics, 34(2), 91-111. https://doi.org/10.1016/j.ajsl.2018.06.006
Middleton, F. (2019, August 8). The 4 types of reliability in research | definitions & examples. Scribbr. https://tinyurl.com/2xndb7vb
Mui Lim, S., Rodger, S., & Brown, T. (2009). Using Rasch analysis to establish the construct validity of rehabilitation assessment tools. International Journal of Therapy and Rehabilitation, 16(5), 251-260. https://doi.org/10.12968/ijtr.2009.16.5.42102
Olson, K. (2010). An examination of questionnaire evaluation by expert reviewers. Field Methods, 22(4), 295-318. https://doi.org/10.1177/1525822X10379795
Ongena, Y. P., Haan, M., Yakar, D., & Kwee, T. C. (2020). Patients’ views on the implementation of artificial intelligence in radiology: development and validation of a standardized questionnaire. European Radiology, 30, 1033-1040. https://doi.org/10.1007/s00330-019-06486-0
Osborne, J. W. (2015). What is rotating in exploratory factor analysis? Practical Assessment, Research and Evaluation, 20(2), Article 2. https://doi.org/10.7275/hb2g-m060
Pataranutaporn, P., Danry, V., Leong, J., Punpongsanon, P., Novy, D., Maes, P., & Sra, M. (2021). AI-generated characters for supporting personalized learning and well-being. Nature Machine Intelligence, 3(12), 1013-1022. https://doi.org/10.1038/s42256-021-00417-9
Sagarika, R. H., Kandakatla, R., & Gulhane, A. (2021). Role of learning analytics to evaluate formative assessments: Using a data driven approach to inform changes in teaching practices. Journal of Engineering Education Transformations, 34, 550-556. https://doi.org/10.16920/jeet/2021/v34i0/157212
Sangapu, I. (2018). Artificial intelligence in education - From a teacher and a student perspective. SSRN. https://doi.org/10.2139/ssrn.3372914
Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18, Article 54. https://doi.org/10.1186/s41239-021-00292-9
Shrestha, N. (2021). Factor analysis as a tool for survey analysis. American Journal of Applied Mathematics and Statistics, 9(1), 4-11. https://doi.org/10.12691/ajams-9-1-2
Tan Ai Lin, D., Ganapathy, M., & Kaur, M. (2018). Kahoot! It: Gamification in higher education. Pertanika Journal of Social Sciences & Humanities, 26(1), 565-582. https://tinyurl.com/bd6mawzd
Tomé-Fernández, M., Fernández-Leyva, C., & Olmedo-Moreno, E. M. (2020). Exploratory and confirmatory factor analysis of the social skills scale for young immigrants. Sustainability, 12(17), Article 6897. https://doi.org/10.3390/SU12176897
Wright, B. D., & Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 8(3), 370-371. https://www.rasch.org/rmt/rmt83b.htm
Yan, Q., Li, D., Yin, X., Jiang, N., Sun, N., Luo, Q., Pang, X., Fan, L., & Gong, Y. (2022). Development and validation of a maternal anxiety for neonatal jaundice scale in China. BMC Psychiatry, 22, Article 526. https://doi.org/10.1186/s12888-022-04161-1
Zach. (2021). What is face validity? (definition & examples). STATOLOGY. https://www.statology.org/face-validity/
Zhang, X. (2006). Factor analysis of public clients' best-value objective in public-privately partnered infrastructure projects. Journal of Construction Engineering and Management, 132(9), 956-965. https://doi.org/10.1061/(ASCE)0733-9364(2006)132:9(956)
Zhu, J., & Ren, C. (2022). Analysis of the Effect of Artificial Intelligence on Role Cognition in the Education System. Occupational Therapy International, 2022, Article 1781662. (Retraction published December 20, 2023, Occupational Therapy International, 2023, Article 9860617). https://doi.org/10.1155/2022/1781662
Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442. https://doi.org/10.1037/0033-2909.99.3.432