logo logo International Journal of Educational Methodology

IJEM is a leading, peer-reviewed, open access, research journal that provides an online forum for studies in education, by and for scholars and practitioners, worldwide.

Subscribe to

Receive Email Alerts

for special events, calls for papers, and professional development opportunities.

Subscribe

Publisher (HQ)

RHAPSODE
Eurasian Society of Educational Research
College House, 2nd Floor 17 King Edwards Road, Ruislip, London, HA4 7AE, UK
RHAPSODE
Headquarters
College House, 2nd Floor 17 King Edwards Road, Ruislip, London, HA4 7AE, UK
Research Article

A High-Stakes Approach to Response Time Effort in Low-Stakes Assessment

Munevver Ilgun Dibek

Response times are one of the important sources that provide information about the performance of individuals during a test process. The main purpose .

R

Response times are one of the important sources that provide information about the performance of individuals during a test process. The main purpose of this study is to show that survival models can be used in educational data. Accordingly, data sets of items measuring literacy, numeracy and problem-solving skills of the countries participating in Round 3 of the Programme for the International Assessment of Adult Competencies were used. Accelerated failure time models have been analyzed for each country and domain.  As a result of the analysis of the models in which various covariates are included as independent variables, and response time for giving correct answers is included as a dependent variable, it was found the associations between the covariates and response time for giving correct answers were concluded to vary from one domain to another or from one country to another. The results obtained from the present study have provided the educational stakeholders and practitioners with valuable information.

Keywords: Accelerated failure time model, PIAAC, response time, survival analysis.

cloud_download PDF
Cite
Article Metrics
Views
333
Download
914
Citations
Crossref
2

Scopus
0

References

Anghel, B., & Balart, P. (2017). Non-cognitive skills and individual earnings: New evidence from PIAAC. Journal of the Spanish Economic Association, 8(1), 417–473. https://doi.org/10.1007/s13209-017-0165-x

Balart, P., & Oosterveen, M. (2019). Females show more sustained performance during test-taking than males. Nature Communications, 10(1), 3798. https://doi.org/10.1038/s41467-019-11691-y

Bekalo, D. B. (2019). Modeling determinants of time-to-circumcision of girls: a comparison of various parametric shared frailty models. Health Services and Outcomes Research Methodology, 19, 145–174. https://doi.org/10.1007/s10742-019-00199-z

Chernick, M. R., & Friis, R. H. (2003). Introductory biostatistics for the health sciences: Modern application including bootstrap. John Wiley and Sons.

Cox, D. R. (1972). Regression models and life‐tables. Journal of the Royal Statistical Society, Series B, 34(2), 187‐220. https://www.jstor.org/stable/2985181

De Boeck, P., & Jeon, M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Applied Mathematics and Statistics, 10, 102. https://doi.org/10.3389/fpsyg.2019.00102

Dhoke, A., Kumar, A., & Ghosh, I. (2021). Hazard-based duration approach to pedestrian crossing behavior at signalized intersections. Transportation Research Record. Advance online publication. https://doi.org/10.1177/03611981211003102

Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153.,

Finn, B. (2015). Measuring motivation in low-stakes assessments (Research Report No. RR-15-19). Princeton. https://doi.org/10.1002/ets2.12067  

Fox, J., & Weisberg, S. (2011). An R companion to applied regression. SAGE.

Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-scale Assessments in Education, 5(18), 1-25. https://doi.org/ 10.1186/s40536-017-0051-9

Goldhammer, F., Naumann, J., Stelter, A., Toth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608–626. https://doi.org/10.1037/a0034716

Hart, T., Chaparro, B., & Halcomb, C. (2008). Evaluating websites for older adults: Adherence to senior-friendly guidelines and end-user performance. Behaviour & Information Technology, 27(3), 191–199, https://doi.org/10.1080/01449290600802031

Harvey, P. D., Tibiriçá, L., Kallestrup, P., & Czaja, S. J. (2020). A Computerized functional skills assessment and training program targeting technology based everyday functional skills. Journal of Visualized Experiments, 156, 1-18, https://doi.org/10.3791/60330

Heitz, R. P. (2014). The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Frontiers in Neuroscience, 8(150), 1-19. https://doi.org/10.3389/fnins.2014.00150

Ilgun Dibek, M. (2020). Silent predictors of test disengagement in PIAAC 2012. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 430-450. https://doi.org/10.21031/epod.796626

Kirsch, I. S., Lennon, M. L., von Davier, M., Gonzalez, E. J., & Yamamoto, K. (2013). On the growing importance of international large-scale assessments, in the role of international large-scale assessments: Perspectives from technology, economy, and educational research. Springer. https://doi.org/10.1007/978-94-007-4629- 9_1

Kleinbaum, D. G., & Klein, M. (2005). Survival analysis: A self-learning text. Springer.

Kureková, L., Haita, C., & Beblavý, M. (2013). Conceptualizing low-skillness: A new approach. Sociology, 45(3), 247-266.

Massing, N., & Schneider, S. L. (2017). Degrees of competency: The relationship between educational qualifications and adult skills across countries. Large-scale Assessment in Education, 5(1), 1-6. https://doi.org/10.1186/s40536-017-0041-y

Nardi, M., & Schemper, M. (2003). Comparing Cox and parametric models in clinical studies. Statistics in Medicine, 22, 3597-3610. https://doi.org/10.1002/sim.1592

Novikov, N. A., Nurislamova, Y. M., Zhozhikashvili, N. A., Kalenkovich, E. E., Lapina, A. A., & Chernishev, B. V. (2017). Slow and fast responses: Two mechanisms of trial outcome processing revealed by EEG oscillations. Frontiers in Human Neuroscience. 11, 218. https://doi.org/10.3389/fnhum.2017.00218

Odo, D. M. (2012). Computer familiarity and test performance on a computer-based cloze ESL reading assessment. Teaching English with Technology, 12(3), 18-35. https://eric.ed.gov/?id=EJ1145240

Orbe, J., Ferreira, E., & Nunez-Anton, V. (2002). Comparing proportional hazards and accelerated failure time models for survival analysis. Statistics in Medicine, 21, 3493-3510. https://doi.org/10.1002/sim.1251

Organization for Economic Co-operation and Development. (2013a). OECD skills outlook 2013: First results from the survey of adult skills. OECD Publishing. https://doi.org/10.1787/9789264204256-en

Organization for Economic Co-operation and Development. (2013b). Skilled for life? Key findings from the survey of adult skills, OECD Publishing. https://bit.ly/2YxcaIN

Organization for Economic Co-operation and Development. (2013c). PISA 2012 Results: Ready to learn: Students’ engagement, drive and self-beliefs (Volume III), PISA. OECD Publishing. https://bit.ly/3Fwyljo

Organization for Economic Co-operation and Development. (2015). Adults, computers and problem solving: What’s the problem? OECD Publishing. https://doi.org/10.1787/9789264236844-en

Organization for Economic Co-operation and Development. (2019a). Beyond Proficiency: Using log files to understand respondent behaviour in the survey of adult skills, OECD Skills Studies, OECD Publishing. https://doi.org/10.1787/0b1414ed-en

Organization for Economic Co-operation and Development. (2019b). Technical report of the survey of adult skills (PIAAC), OECD Publishing. https://bit.ly/2YzyfGU

Organization for Economic Co-operation and Development. (2019c). The survey of adult skills: Reader’s companion (3rd ed.). OECD Publishing. https://doi.org/10.1787/f70238c7-en

Organization for Economic Co-operation and Development. (2020). Education at a glance 2020: OECD indicators, OECD Publishing. https://doi.org/10.1787/69096873-en

O’Neil, H. F., Sugrue, B., & Baker, E. L. (1995). Effects of motivational interventions on the national assessment of educational progress mathematics performance. Educational Assessment, 3(2), 135–157. https://doi.org/10.1207/s15326977ea0302_2

Pomplun, M., Frey, S., & Becker, D. (2002). The score equivalence of paper-and-pencil and computerized versions of a speeded test of reading comprehension. Educational and Psychological Measurement, 62(2), 337-354. https://doi.org/10.1177/0013164402062002009

Qiao, Y., Labi, S., & Fricker, J. D. (2019). Hazard-based duration models for predicting actual duration of highway projects using nonparametric and parametric survival analysis. Journal of Management in Engineering, 35(6), 1-16. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000700

Qui, J. (2009). Comparison of proportional hazards and accelerated failure time models [Unpublished master’s thesis]. University of Saskatchewan.

Quinn, D. M., & Cooc, N. (2015). Science achievement gaps by gender and race/ethnicity in elementary and middle school trends and predictors. Educational Researcher, 44(6), 336–346. https://doi.org/10.3102/0013189X15598539

Ranger, J., Kuhn, J. T., & Ortner, T. M. (2020). Modeling responses and response times in tests with the hierarchical model and the three-parameter lognormal distribution. Educational and Psychological Measurement, 80(6), 1059–1089. https://doi.org/10.1177/0013164420908916

Ranger, J., & Ortner, T. (2012). A latent trait model for response times on tests employing the proportional hazards model. British Journal of Mathematical and Statistical Psychology, 65(2), 334–349. https://doi.org/10.1111/j.2044-8317.2011.02032.x

Ratcliff, R., Thapar, A., Gomez, P., & McKoon, G. (2004). A diffusion model analysis of the effects of aging in the lexical-decision task. Psychology and Aging, 19(2), 278–289. https://doi.org/10.1037/0882-7974.19.2.278

Reder, S. (1998). In MC. Smith (Ed.). Literacy for the 21st century: Research, policy, practice and the National Adult Literacy Survey. Greenwood Press.

Rouder, J., Province, J., Morey, R., Gomez, P., & Heathkote, A. (2015). The lognormal race: a cognitive- process model of choice and latency with desirable psychometric properties. Psychometrika, 80(2), 491–513. https://doi.org/10.1007/s11336-013-9396-3

Sahin, F., & Colvin, K. F. (2020). Enhancing response time thresholds with response behaviors for detecting disengaged examinees. Large-scale Assessment in Education, 8(5). https://doi.org/10.1186/s40536-020-00082-1

Saikia, R., & Barman, M. P. (2017). A review on accelerated failure time models. International Journal of Statistics and Systems, 12(2), 311-322. https://bit.ly/3Btzw0o

Sawaki, Y. (2001). Comparability of conventional and computerized tests of reading in a second language. Language Learning & Technology, 5(2), 38-59. http://dx.doi.org/10125/25127

Scheuermann, F., & Bjornsson, J. (Eds.). (2009). The transition to computer-based assessment. Office for Official Publications of the European Communities.

Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213-232. https://doi.org/10.1111/j.1745-3984.1997.tb00516.x

Singh, R., & Mukhopadhyay, K. (2011). Survival analysis in clinical trials: Basics and must know areas. Perspectives in Clinical Research, 2(4), 145–148. https://doi.org/10.4103/2229-3485.86872

Smith, M. C., Rose, A. D., Ross-Gordon, J., & Smith, T. J. (2015). Adults’ readiness to learn as a predictor of literacy skills. https://bit.ly/3lnVlc9

Sternberg, R. J. (1985). Beyond IQ. A triarchic theory of human intelligence. University Press.

Swindell, W. R. (2009). Accelerated failure time models provide a useful statistical framework for aging research. Experimental gerontology, 44(3), 190-200. https://doi.org/10.1016/j.exger.2008.10.005

van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

van der Maas, H. L. J., Molenaar, D., Maris, G., Kievit, R., & Borsboom, D. (2011). Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological Review, 118(2). 339–356. https://doi.org/10.1037/a0022749  

van Zandt, T. (2002). Analysis of response time distributions. In H. Pashler & J. Wixted (Eds.), Stevens' handbook of experimental psychology: Methodology in experimental psychology (pp. 461–516). John Wiley & Sons Inc.

Wang, C., Fan, Z., Chang, H. H., & Douglas, J. A. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66(1), 144–168. https://doi.org/10.1111/j.2044-8317.2012. 02045.x

Wang, S., & Chen, Y. (2020). Using response times and response accuracy to measure fluency within cognitive diagnosis models. Psychometrika, 85(3), 600–629. https://doi.org/10.1007/s11336-020-09717-2

Weakliem, D. L. (1999). A critique of the Bayesian information criterion for model selection. Sociological Methods & Research, 27(3), 359–397.

Wei, L. J. (1992). The accelerated failure time model: A useful alternative to the Cox regression model in survival analysis, Statistics in Medicine, 11(14-15), 1871-1879. https://doi.org/10.1002/sim.4780111409

Wenger, M., & Gibson, B. (2004). Using hazard functions to assess changes in processing capacity in an attentional cuing paradigm. Journal of Experimental Psychology, 30(4), 708–719. https://doi.org/10.1037/0096-1523.30.4.708

Wheatley-Price, P., Hutton, B., & Clemons, M. (2012). The Mayan Doomsday’s effect on survival outcomes in clinical trials. Canadian Medical Association Journal, 184(18), 2021–2022. https://doi.org/10.1503/cmaj.121616

Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://doi.org/10.1207/s15326977ea1001_1

Wise, S. L., & DeMars, C. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15(1), 27–41. https://doi.org/10.1080/10627191003673216

Wise, S. L., & Kong, X. J. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2

...