A High-Stakes Approach to Response Time Effort in Low-Stakes Assessment
Response times are one of the important sources that provide information about the performance of individuals during a test process. The main purpose .
- Pub. date: November 15, 2021
- Pages: 571-586
- 333 Downloads
- 914 Views
- 2 Citations
Response times are one of the important sources that provide information about the performance of individuals during a test process. The main purpose of this study is to show that survival models can be used in educational data. Accordingly, data sets of items measuring literacy, numeracy and problem-solving skills of the countries participating in Round 3 of the Programme for the International Assessment of Adult Competencies were used. Accelerated failure time models have been analyzed for each country and domain. As a result of the analysis of the models in which various covariates are included as independent variables, and response time for giving correct answers is included as a dependent variable, it was found the associations between the covariates and response time for giving correct answers were concluded to vary from one domain to another or from one country to another. The results obtained from the present study have provided the educational stakeholders and practitioners with valuable information.
Keywords: Accelerated failure time model, PIAAC, response time, survival analysis.
References
Anghel, B., & Balart, P. (2017). Non-cognitive skills and individual earnings: New evidence from PIAAC. Journal of the Spanish Economic Association, 8(1), 417–473. https://doi.org/10.1007/s13209-017-0165-x
Balart, P., & Oosterveen, M. (2019). Females show more sustained performance during test-taking than males. Nature Communications, 10(1), 3798. https://doi.org/10.1038/s41467-019-11691-y
Bekalo, D. B. (2019). Modeling determinants of time-to-circumcision of girls: a comparison of various parametric shared frailty models. Health Services and Outcomes Research Methodology, 19, 145–174. https://doi.org/10.1007/s10742-019-00199-z
Chernick, M. R., & Friis, R. H. (2003). Introductory biostatistics for the health sciences: Modern application including bootstrap. John Wiley and Sons.
Cox, D. R. (1972). Regression models and life‐tables. Journal of the Royal Statistical Society, Series B, 34(2), 187‐220. https://www.jstor.org/stable/2985181
De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Applied Mathematics and Statistics, 10, 102. https://doi.org/10.3389/fpsyg.2019.00102
Dhoke, A., Kumar, A., & Ghosh, I. (2021). Hazard-based duration approach to pedestrian crossing behavior at signalized intersections. Transportation Research Record. Advance online publication. https://doi.org/10.1177/03611981211003102
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153.,
Finn, B. (2015). Measuring motivation in low-stakes assessments (Research Report No. RR-15-19). Princeton. https://doi.org/10.1002/ets2.12067
Fox, J., & Weisberg, S. (2011). An R companion to applied regression. SAGE.
Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-scale Assessments in Education, 5(18), 1-25. https://doi.org/ 10.1186/s40536-017-0051-9
Goldhammer, F., Naumann, J., Stelter, A., Toth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608–626. https://doi.org/10.1037/a0034716
Hart, T., Chaparro, B., & Halcomb, C. (2008). Evaluating websites for older adults: Adherence to senior-friendly guidelines and end-user performance. Behaviour & Information Technology, 27(3), 191–199, https://doi.org/10.1080/01449290600802031
Harvey, P. D., Tibiriçá, L., Kallestrup, P., & Czaja, S. J. (2020). A Computerized functional skills assessment and training program targeting technology based everyday functional skills. Journal of Visualized Experiments, 156, 1-18, https://doi.org/10.3791/60330
Heitz, R. P. (2014). The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Frontiers in Neuroscience, 8(150), 1-19. https://doi.org/10.3389/fnins.2014.00150
Ilgun Dibek, M. (2020). Silent predictors of test disengagement in PIAAC 2012. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 430-450. https://doi.org/10.21031/epod.796626
Kirsch, I. S., Lennon, M. L., von Davier, M., Gonzalez, E. J., & Yamamoto, K. (2013). On the growing importance of international large-scale assessments, in the role of international large-scale assessments: Perspectives from technology, economy, and educational research. Springer. https://doi.org/10.1007/978-94-007-4629- 9_1
Kleinbaum, D. G., & Klein, M. (2005). Survival analysis: A self-learning text. Springer.
Kureková, L., Haita, C., & Beblavý, M. (2013). Conceptualizing low-skillness: A new approach. Sociology, 45(3), 247-266.
Massing, N., & Schneider, S. L. (2017). Degrees of competency: The relationship between educational qualifications and adult skills across countries. Large-scale Assessment in Education, 5(1), 1-6. https://doi.org/10.1186/s40536-017-0041-y
Nardi, M., & Schemper, M. (2003). Comparing Cox and parametric models in clinical studies. Statistics in Medicine, 22, 3597-3610. https://doi.org/10.1002/sim.1592
Novikov, N. A., Nurislamova, Y. M., Zhozhikashvili, N. A., Kalenkovich, E. E., Lapina, A. A., & Chernishev, B. V. (2017). Slow and fast responses: Two mechanisms of trial outcome processing revealed by EEG oscillations. Frontiers in Human Neuroscience. 11, 218. https://doi.org/10.3389/fnhum.2017.00218
Odo, D. M. (2012). Computer familiarity and test performance on a computer-based cloze ESL reading assessment. Teaching English with Technology, 12(3), 18-35. https://eric.ed.gov/?id=EJ1145240
Orbe, J., Ferreira, E., & Nunez-Anton, V. (2002). Comparing proportional hazards and accelerated failure time models for survival analysis. Statistics in Medicine, 21, 3493-3510. https://doi.org/10.1002/sim.1251
Organization for Economic Co-operation and Development. (2013a). OECD skills outlook 2013: First results from the survey of adult skills. OECD Publishing. https://doi.org/10.1787/9789264204256-en
Organization for Economic Co-operation and Development. (2013b). Skilled for life? Key findings from the survey of adult skills, OECD Publishing. https://bit.ly/2YxcaIN
Organization for Economic Co-operation and Development. (2013c). PISA 2012 Results: Ready to learn: Students’ engagement, drive and self-beliefs (Volume III), PISA. OECD Publishing. https://bit.ly/3Fwyljo
Organization for Economic Co-operation and Development. (2015). Adults, computers and problem solving: What’s the problem? OECD Publishing. https://doi.org/10.1787/9789264236844-en
Organization for Economic Co-operation and Development. (2019a). Beyond Proficiency: Using log files to understand respondent behaviour in the survey of adult skills, OECD Skills Studies, OECD Publishing. https://doi.org/10.1787/0b1414ed-en
Organization for Economic Co-operation and Development. (2019b). Technical report of the survey of adult skills (PIAAC), OECD Publishing. https://bit.ly/2YzyfGU
Organization for Economic Co-operation and Development. (2019c). The survey of adult skills: Reader’s companion (3rd ed.). OECD Publishing. https://doi.org/10.1787/f70238c7-en
Organization for Economic Co-operation and Development. (2020). Education at a glance 2020: OECD indicators, OECD Publishing. https://doi.org/10.1787/69096873-en
O’Neil, H. F., Sugrue, B., & Baker, E. L. (1995). Effects of motivational interventions on the national assessment of educational progress mathematics performance. Educational Assessment, 3(2), 135–157. https://doi.org/10.1207/s15326977ea0302_2
Pomplun, M., Frey, S., & Becker, D. (2002). The score equivalence of paper-and-pencil and computerized versions of a speeded test of reading comprehension. Educational and Psychological Measurement, 62(2), 337-354. https://doi.org/10.1177/0013164402062002009
Qiao, Y., Labi, S., & Fricker, J. D. (2019). Hazard-based duration models for predicting actual duration of highway projects using nonparametric and parametric survival analysis. Journal of Management in Engineering, 35(6), 1-16. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000700
Qui, J. (2009). Comparison of proportional hazards and accelerated failure time models [Unpublished master’s thesis]. University of Saskatchewan.
Quinn, D. M., & Cooc, N. (2015). Science achievement gaps by gender and race/ethnicity in elementary and middle school trends and predictors. Educational Researcher, 44(6), 336–346. https://doi.org/10.3102/0013189X15598539
Ranger, J., Kuhn, J. T., & Ortner, T. M. (2020). Modeling responses and response times in tests with the hierarchical model and the three-parameter lognormal distribution. Educational and Psychological Measurement, 80(6), 1059–1089. https://doi.org/10.1177/0013164420908916
Ranger, J., & Ortner, T. (2012). A latent trait model for response times on tests employing the proportional hazards model. British Journal of Mathematical and Statistical Psychology, 65(2), 334–349. https://doi.org/10.1111/j.2044-8317.2011.02032.x
Ratcliff, R., Thapar, A., Gomez, P., & McKoon, G. (2004). A diffusion model analysis of the effects of aging in the lexical-decision task. Psychology and Aging, 19(2), 278–289. https://doi.org/10.1037/0882-7974.19.2.278
Reder, S. (1998). In MC. Smith (Ed.). Literacy for the 21st century: Research, policy, practice and the National Adult Literacy Survey. Greenwood Press.
Rouder, J., Province, J., Morey, R., Gomez, P., & Heathkote, A. (2015). The lognormal race: a cognitive- process model of choice and latency with desirable psychometric properties. Psychometrika, 80(2), 491–513. https://doi.org/10.1007/s11336-013-9396-3
Sahin, F., & Colvin, K. F. (2020). Enhancing response time thresholds with response behaviors for detecting disengaged examinees. Large-scale Assessment in Education, 8(5). https://doi.org/10.1186/s40536-020-00082-1
Saikia, R., & Barman, M. P. (2017). A review on accelerated failure time models. International Journal of Statistics and Systems, 12(2), 311-322. https://bit.ly/3Btzw0o
Sawaki, Y. (2001). Comparability of conventional and computerized tests of reading in a second language. Language Learning & Technology, 5(2), 38-59. http://dx.doi.org/10125/25127
Scheuermann, F., & Bjornsson, J. (Eds.). (2009). The transition to computer-based assessment. Office for Official Publications of the European Communities.
Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213-232. https://doi.org/10.1111/j.1745-3984.1997.tb00516.x
Singh, R., & Mukhopadhyay, K. (2011). Survival analysis in clinical trials: Basics and must know areas. Perspectives in Clinical Research, 2(4), 145–148. https://doi.org/10.4103/2229-3485.86872
Smith, M. C., Rose, A. D., Ross-Gordon, J., & Smith, T. J. (2015). Adults’ readiness to learn as a predictor of literacy skills. https://bit.ly/3lnVlc9
Sternberg, R. J. (1985). Beyond IQ. A triarchic theory of human intelligence. University Press.
Swindell, W. R. (2009). Accelerated failure time models provide a useful statistical framework for aging research. Experimental gerontology, 44(3), 190-200. https://doi.org/10.1016/j.exger.2008.10.005
van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z
van der Maas, H. L. J., Molenaar, D., Maris, G., Kievit, R., & Borsboom, D. (2011). Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological Review, 118(2). 339–356. https://doi.org/10.1037/a0022749
van Zandt, T. (2002). Analysis of response time distributions. In H. Pashler & J. Wixted (Eds.), Stevens' handbook of experimental psychology: Methodology in experimental psychology (pp. 461–516). John Wiley & Sons Inc.
Wang, C., Fan, Z., Chang, H. H., & Douglas, J. A. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66(1), 144–168. https://doi.org/10.1111/j.2044-8317.2012. 02045.x
Wang, S., & Chen, Y. (2020). Using response times and response accuracy to measure fluency within cognitive diagnosis models. Psychometrika, 85(3), 600–629. https://doi.org/10.1007/s11336-020-09717-2
Weakliem, D. L. (1999). A critique of the Bayesian information criterion for model selection. Sociological Methods & Research, 27(3), 359–397.
Wei, L. J. (1992). The accelerated failure time model: A useful alternative to the Cox regression model in survival analysis, Statistics in Medicine, 11(14-15), 1871-1879. https://doi.org/10.1002/sim.4780111409
Wenger, M., & Gibson, B. (2004). Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. Journal of Experimental Psychology, 30(4), 708–719. https://doi.org/10.1037/0096-1523.30.4.708
Wheatley-Price, P., Hutton, B., & Clemons, M. (2012). The Mayan Doomsday’s effect on survival outcomes in clinical trials. Canadian Medical Association Journal, 184(18), 2021–2022. https://doi.org/10.1503/cmaj.121616
Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://doi.org/10.1207/s15326977ea1001_1
Wise, S. L., & DeMars, C. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15(1), 27–41. https://doi.org/10.1080/10627191003673216
Wise, S. L., & Kong, X. J. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2