logo logo International Journal of Educational Methodology

IJEM is a leading, peer-reviewed, open access, research journal that provides an online forum for studies in education, by and for scholars and practitioners, worldwide.

Subscribe to

Receive Email Alerts

for special events, calls for papers, and professional development opportunities.

Subscribe

Publisher (HQ)

RHAPSODE LTD
Eurasian Society of Educational Research
College House, 2nd Floor 17 King Edwards Road, Ruislip, London, UK. HA4 7AE
RHAPSODE LTD
Headquarters
College House, 2nd Floor 17 King Edwards Road, Ruislip, London, UK. HA4 7AE
artificial intelligence item measurement reliability test validity test

Development and Validation of Instruments for Assessing the Impact of Artificial Intelligence on Students in Higher Education

Andie Tangonan Capinding

The role of artificial intelligence (AI) in education remains incompletely understood, demanding further evaluation and the creation of robust assessm.


  • Pub. date: May 15, 2024
  • Online Pub. date: February 06, 2024
  • Pages: 197-211
  • 72 Downloads
  • 206 Views
  • 0 Citations
T

The role of artificial intelligence (AI) in education remains incompletely understood, demanding further evaluation and the creation of robust assessment tools. Despite previous attempts to measure AI's impact in education, existing studies have limitations. This research aimed to develop and validate an assessment instrument for gauging AI effects in higher education. Employing various analytical methods, including Exploratory Factor Analysis, Confirmatory Factor Analysis, and Rasch Analysis, the initial 70-item instrument covered seven constructs. Administered to 635 students at Nueva Ecija University of Science and Technology – Gabaldon campus, content validity was assessed using the Lawshe method. After eliminating 19 items through EFA and CFA, Rasch analysis confirmed the construct validity and led to the removal of three more items. The final 48-item instrument, categorized into learning experiences, academic performance, career guidance, motivation, self-reliance, social interactions, and AI dependency, emerged as a valid and reliable tool for assessing AI's impact on higher education, especially among college students.

Keywords: Artificial intelligence, item measurement, reliability test, validity test.

cloud_download PDF
Cite
Article Metrics
Views
72
Download
206
Citations
Crossref
0

Scopus
0

References

Akour, M. M. (2022). Rasch rating scale analysis of the survey of attitudes toward statistics. EURASIA Journal of Mathematics, Science and Technology Education, 18(12), Article em2190. https://doi.org/10.29333/ejmste/12646

Albano, T. (2020). Introduction to educational and psychological measurement using R. Thetaminusb. http://tinyurl.com/2xcethux

Awang, Z. (2012). Structural equation modelling using AMOS graphic. UiTM Press.

Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79-86. https://doi.org/10.1177/07481756135138

Baghaei, P. (2008). The Rasch model as a construct validation tool. Rasch Measurement Transactions, 22(1), 1145-1146. http://tinyurl.com/479sxke5

Bhandari, P. (2022). What is face validity? | guide, definition & examples. Scribbr. https://tinyurl.com/ye23n5c6

Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Lawrence Erlbaum Associates.

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.

Booc, N. B., Sobremisana, K., Ybañez, A., Tolosa, R., Ladroma, S. M., & Caparoso, K. M. (2023). Artificial intelligence-powered calculator application usage in mathematics summative assessments. Iconic Research and Engineering Journals, 6(10), 446-474. https://bit.ly/3unOdDQ

Cliff, N. (1988). The eigenvalues-greater-than-one rule and the reliability of components. Psychological Bulletin, 103(2), 276. https://doi.org/10.1037/0033-2909.103.2.276

Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis. Erlbaum.

Denecke, K., Abd-Alrazaq, A., & Househ, M. (2021). Artificial intelligence for chatbots in mental health: Opportunities and challenges. In M. Househ, E. Borycki & A. Kushniruk (Eds.), Multiple perspectives on artificial intelligence in healthcare: Opportunities and challenges (pp. 115-128). Springer. https://doi.org/10.1007/978-3-030-67303-1_10

Eltahir, M. E., Alsalhi, N. R., Al-Qatawneh, S., AlQudah, H. A., & Jaradat, M. (2021). The impact of game-based learning (GBL) on students’ motivation, engagement and academic performance on an Arabic language grammar course in higher education. Education and Information Technologies, 26, 3251-3278. https://doi.org/10.1007/s10639-020-10396-w

Field, A. (2013). Discovering statistics using IBM SPSS statistics. Sage.

Frost, J. (2023). Cronbach’s alpha: Definition, calculations & example. Statistics by Jim. http://tinyurl.com/3n55dr77

Gaskination's StatWiki. (n.d.). CFA. https://bit.ly/3Oiw54X

George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Allyn & Bacon.

Gerber, N. L., & Price, J. K. (2018). Measures of function and health-related quality of life. In J. Gallin, F. Ognibene & L. Johnson (Eds.), Principles and practice of clinical research (pp. 303-315). Academic Press. https://doi.org/10.1016/B978-0-12-849905-4.00021-6

Goodwyn, F. (2012). Question number two: how many factors? (ED529100). ERIC. https://eric.ed.gov/?id=ED529100

Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis (5th ed.). Prentice-Hall

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis. Pearson University Press.

Hassan, A. (2011). Kesahan dan kebolehpercayaan item penilaian pembimbing dalam pembelajaran berasaskan kerja (PBK) menggunakan model pengukuran Rasch [Validity and trustworthiness of supervisor assessment items in work-based learning (PBK) using the Rasch measurement model]. USM, Psychometrics Centre, MIMOS & Malaysian Examination Syndicate, MOE.

Hsu, T. C., & Chen, M. S. (2022). The engagement of students when learning to use a personal audio classifier to control robot cars in a computational thinking board game. Research and Practice in Technology Enhanced Learning, 17, Article 27. https://doi.org/10.1186/s41039-022-00202-1

Hwang, G. J., Sung, H. Y., Chang, S. C., & Huang, X. C. (2020). A fuzzy expert system-based adaptive learning approach to improving students’ learning performances by considering affective and cognitive factors. Computers and Education: Artificial Intelligence, 1, Article 100003. https://doi.org/10.1016/j.caeai.2020.100003

Irons, A., & Elkington, S. (2007). Enhancing learning through formative assessment and feedback. Routledge. https://doi.org/10.4324/9780203934333

Jindal, A., & Bansal, M. (2020). Knowledge and education about artificial intelligence among medical students from teaching institutions of India: A brief survey [version 1]. MedEdPublish, 9, Article 200. https://doi.org/10.15694/mep.2020.000200.1

Kairu, C. (2020). Students’ attitude towards the use of artificial intelligence and machine learning to measure classroom engagement activities. In Proceedings of EdMedia+ Innovate Learning (pp. 793-802). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/p/217382.

Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36. https://doi.org/10.1007/BF02291575

Kaiser, H. F. (1960). Varimax solution for primary mental abilities. Psychometrika, 25(2), 153-158. https://doi.org/10.1007/BF02288578

Kang, M., & Im, T. (2013). Factors of learner–instructor interaction which predict perceived learning outcomes in online learning environment. Journal of Computer Assisted Learning, 29(3), 292-301. https://doi.org/10.1111/jcal.12005

Kline, R. B. (2023). Principles and practice of structural equation modeling. Guilford Publications.

Knekta, E., Runyon, C., & Eddy, S. (2019). One size doesn’t fit all: Using factor analysis to gather validity evidence when using surveys in your research. CBE—Life Sciences Education, 18(1), 1-17. https://doi.org/10.1187/cbe.18-04-0064

Kooli, C. (2023). Chatbots in education and research: A critical examination of ethical implications and solutions. Sustainability, 15(7), 5614. https://doi.org/10.3390/su15075614

Lawshe, C. H. (1975). A quantitative approach to con-tent validity. Personnel Psychology, 28, 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

Lennartz, S., Dratsch, T., Zopfs, D., Persigehl, T., Maintz, D., Große Hokamp, N., & Pinto dos Santos, D. (2021). Use and control of artificial intelligence in patients across the medical workflow: single-center questionnaire study of patient perspectives. Journal of Medical Internet Research, 23(2), Article 24221. https://doi.org/10.2196/24221

Linacre, J. M. (2007). A user’s guide to WINSTEPS Rasch Model computer programs. MESA Press

Linacre, J. M. (2012). A user's guide to WINSTEPS® MINISTEP Rasch-model computer programs. Winsteps.Com. https://www.winsteps.com/winman/copyright.htm

Ma, Y., & Siau, K. L. (2018). Artificial intelligence impacts on higher education. In Proceedings of the Thirteenth Midwest Association for Information Systems Conference (pp. 1-5). Association for Information Systems Electronic Library (AISeL). https://aisel.aisnet.org/mwais2018/42

Malhotra, A. (2020, November 3). The promise of artificial intelligence (AI) in education. Linkedin. http://tinyurl.com/22ys5wab

Masero, R. (2023). Evolving Education: The Impact of AI and VR technology on the future of learning. Learning Industry. http://tinyurl.com/4s9uj9nx

Maskey, R., Fei, J., & Nguyen, H. O. (2018). Use of exploratory factor analysis in maritime research. The Asian Journal of Shipping and Logistics, 34(2), 91-111. https://doi.org/10.1016/j.ajsl.2018.06.006

Middleton, F. (2019, August 8). The 4 types of reliability in research | definitions & examples. Scribbr. https://tinyurl.com/2xndb7vb

Mui Lim, S., Rodger, S., & Brown, T. (2009). Using Rasch analysis to establish the construct validity of rehabilitation assessment tools. International Journal of Therapy and Rehabilitation, 16(5), 251-260. https://doi.org/10.12968/ijtr.2009.16.5.42102

Olson, K. (2010). An examination of questionnaire evaluation by expert reviewers. Field Methods, 22(4), 295-318. https://doi.org/10.1177/1525822X10379795

Ongena, Y. P., Haan, M., Yakar, D., & Kwee, T. C. (2020). Patients’ views on the implementation of artificial intelligence in radiology: development and validation of a standardized questionnaire. European Radiology, 30, 1033-1040. https://doi.org/10.1007/s00330-019-06486-0

Osborne, J. W. (2015). What is rotating in exploratory factor analysis? Practical Assessment, Research and Evaluation, 20(2), Article 2. https://doi.org/10.7275/hb2g-m060

Pataranutaporn, P., Danry, V., Leong, J., Punpongsanon, P., Novy, D., Maes, P., & Sra, M. (2021). AI-generated characters for supporting personalized learning and well-being. Nature Machine Intelligence, 3(12), 1013-1022. https://doi.org/10.1038/s42256-021-00417-9

Sagarika, R. H., Kandakatla, R., & Gulhane, A. (2021). Role of learning analytics to evaluate formative assessments: Using a data driven approach to inform changes in teaching practices. Journal of Engineering Education Transformations, 34, 550-556. https://doi.org/10.16920/jeet/2021/v34i0/157212

Sangapu, I. (2018). Artificial intelligence in education - From a teacher and a student perspective. SSRN. https://doi.org/10.2139/ssrn.3372914

Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18, Article 54. https://doi.org/10.1186/s41239-021-00292-9

Shrestha, N. (2021). Factor analysis as a tool for survey analysis. American Journal of Applied Mathematics and Statistics, 9(1), 4-11. https://doi.org/10.12691/ajams-9-1-2

Tan Ai Lin, D., Ganapathy, M., & Kaur, M. (2018). Kahoot! It: Gamification in higher education. Pertanika Journal of Social Sciences & Humanities, 26(1), 565-582. https://tinyurl.com/bd6mawzd

Tomé-Fernández, M., Fernández-Leyva, C., & Olmedo-Moreno, E. M. (2020). Exploratory and confirmatory factor analysis of the social skills scale for young immigrants. Sustainability, 12(17), Article 6897. https://doi.org/10.3390/SU12176897

Wright, B. D., & Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 8(3), 370-371. https://www.rasch.org/rmt/rmt83b.htm

Yan, Q., Li, D., Yin, X., Jiang, N., Sun, N., Luo, Q., Pang, X., Fan, L., & Gong, Y. (2022). Development and validation of a maternal anxiety for neonatal jaundice scale in China. BMC Psychiatry, 22, Article 526. https://doi.org/10.1186/s12888-022-04161-1

Zach. (2021). What is face validity? (definition & examples). STATOLOGY. https://www.statology.org/face-validity/

Zhang, X. (2006). Factor analysis of public clients' best-value objective in public-privately partnered infrastructure projects. Journal of Construction Engineering and Management, 132(9), 956-965. https://doi.org/10.1061/(ASCE)0733-9364(2006)132:9(956)

Zhu, J., & Ren, C. (2022). Analysis of the Effect of Artificial Intelligence on Role Cognition in the Education System. Occupational Therapy International, 2022, Article 1781662. (Retraction published December 20, 2023, Occupational Therapy International, 2023, Article 9860617). https://doi.org/10.1155/2022/1781662

Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442. https://doi.org/10.1037/0033-2909.99.3.432

...