logo logo International Journal of Educational Methodology

IJEM is a leading, peer-reviewed, open access, research journal that provides an online forum for studies in education, by and for scholars and practitioners, worldwide.

Subscribe to

Receive Email Alerts

for special events, calls for papers, and professional development opportunities.

Subscribe

Publisher (HQ)

RHAPSODE LTD
Eurasian Society of Educational Research
College House, 2nd Floor 17 King Edwards Road, Ruislip, London, UK. HA4 7AE
RHAPSODE LTD
Headquarters
College House, 2nd Floor 17 King Edwards Road, Ruislip, London, UK. HA4 7AE
item analysis goodman kruskal gamma somers d jonckheere terpstra test pearson correlation

Goodman–Kruskal gamma and Dimension-Corrected Gamma in Educational Measurement Settings

Jari Metsämuuronen

Although Goodman–Kruskal gamma (G) is used relatively rarely it has promising potential as a coefficient of association in educational settings..

A

Although Goodman–Kruskal gamma (G) is used relatively rarely it has promising potential as a coefficient of association in educational settings.  Characteristics of G are studied in three sub-studies related to educational measurement settings. G appears to be unexpectedly appealing as an estimator of association between an item and a score because it strictly indicates the probability to get a correct answer in the test item given the score, and it accurately produces perfect latent association irrespective of distributions, degrees of freedom, number of tied pairs and tied values in the variables, or the difficulty levels in the items. However, it underestimates the association in an obvious manner when the number of categories in the item is more than four. Towards this, a dimension-corrected G (G2) is proposed and its characteristics are studied. Both G and G2 appear to be promising alternatives in measurement modelling settings, G with binary items and G2 with binary, polytomous and mixed datasets.

Keywords: Item analysis, Goodman–Kruskal gamma, Somers D, Jonckheere–Terpstra test, Pearson correlation.

cloud_download PDF
Cite
Article Metrics
Views
867
Download
794
Citations
Crossref
9

Scopus

References

Agresti, A. (2010). Analysis of ordinal categorical data (2nd ed.). Wiley.

Arbuthnott, J. (1710). An argument for divine providence, taken from the constant regularity observed in the births of both sexes. Philosophical Transactions of the Royal Society of London, 27(325–336), 186–190. http://doi.org/10.1098/rstl.1710.0011

Aslan, S., & Aybek, B. (2020). Testing the effectiveness of interdisciplinary curriculum-based multicultural education on tolerance and critical thinking skill. International Journal of Educational Methodology, 6(1), 43–55. https://doi.org/10.12973/ijem.6.1.43

Bai, J., & Wei, L.-L. (2009). A new method of attribute reduction based on gamma coefficient. In S.-M. Zhou & W. Wang, GCIS 2009. 2009 WRI Global Congress on Intelligent Systems (pp. 370–373). IEEE Computer Society. https://doi.org/10.1109/GCIS.2009.212

Bravais, A. (1844). Analyse Mathematique. Sur les probabilités des erreurs de situation d'un point [Mathematical analysis. On the probabilities of the point errors]. Imprimerie Royale.

Breslow, N. (1970). A generalized Kruskal–Wallis test for comparing K samples subject to unequal patterns of censorship. BiometricsBiometrika, 57(3), 579–594. http://doi.org/10.1093/biomet/57.3.579

Byrne, B. M. (2016). Structural Equation Modeling with AMOS. Basic concepts, applications, and programming (3rd ed.). Routledge.

Cheng, Y., Yuan, K.-H., & Liu, C. (2012). Comparison of reliability measures under factor analysis and item response theory. Educational and Psychological Measurement, 72(1), 52–67. https://doi.org/10.1177/0013164411407315

Cleff, T. (2019). Applied Statistics and Multivariate Data Analysis for Business and Economics. A Modern Approach Using SPSS, Stata, and Excel. Springer.

Conover, W. J. (1980). Practical nonparametric statistics. Wiley & Sons.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. PsychometricsPsychometrika, 16(3), 297–334. http://doi.org/10.1007/BF02310555

Davis, J. A. (1967). A partial coefficient for Goodman and Kruskal's gamma. Journal of the American Statistical Association, 62(317), 189–193. https://doi.org/10.1080/01621459.1967.10482900

Delil, A., & Ozcan, B. N. (2019). How 8th graders are assessed through tests by mathematics teachers? International Journal of Educational Methodology, 5(3), 479–488. https://doi.org/10.12973/ijem.5.3.479

Drasgow, F. (1986). Polychoric and polyserial correlations. In S. Kotz & N. L. Johnson (Eds.), Encyclopedia of statistical sciences. (Vol. 7, pp. 68–74). John Wiley.

El-Shaarawi, A. H., & Piegorsch, W. W. (2001). Encyclopedia of Environmetrics (Volume 1). John Wiley and Sons.

Finnish National Education Evaluation Centre (2018). National assessment of learning outcomes in mathematics at grade 9 in 2002. Unpublished dataset opened for the re-analysis 18.2.2018. Finnish National Education Evaluation Centre.

Forthmann, B., Förster, N., Schütze, B., Hebbecker, K., Flessner, J., Peters, M. T., & Souvignier, E. (2020). How much g is in the distractor? Re-thinking item-analysis of multiple-choice items. Journal of Intelligence8(1), 1-36. https://doi.org/10.3390/jintelligence8010011

Freeman, L. C. (1986). Order-based statistics and monotonicity: A family of ordinal measures of association. Journal of Mathematical Sociology, 12(1), 49–69. https://doi.org/10.1080/0022250X.1986.9990004

Galton, F. (1889). Kinship and correlation. Statistical Science4(2), 81–86. http://doi.org/10.1214/ss/1177012581

Gehan, E. A. (1965). A generalized Wilcoxon test for comparing arbitrarily single-censored samples. BiometricsBiometrika, 52(1–2), 203–233. http://doi.org/10.1093/biomet/52.1-2.203

Gini, C. (1912). Variabilità e mutabilità. Contributo allo studio delle distribuzioni e dellerelazioni statistiche [Variability and mutability. Contribution to the study of distributions and statistical relationships]. Bologna.

Göktaş, A., & İşçi., O. A. (2011). Comparison of the most commonly used measures of association for doubly ordered square contingency tables via simulation. Methodological Notebooks / Metodološki zvezki8(1), 17–37.

Gonzalez, R., & Nelson, T. O. (1996). Measuring ordinal association in situations that contain tied scoresPsychological Bulletin, 119(1), 159–165. https://doi.org/10.1037/0033-2909.119.1.159

Good, K. (2015). Investigating relationships between educational technology use and other instructional elements using "big data" in higher education [Doctoral dissertation, Iowa State University]. Iowa State University Digital Repository. https://lib.dr.iastate.edu/etd/14854

Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49(268), 732–764. http://doi.org/10.1080/01621459.1954.10501231

Goodman, L. A., & Kruskal, W. H. (1979). Measures of association for cross classification. Springer-Verlag.

Green S. B., & Yang Y. (2009). Commentary on coefficient alpha: A cautionary tale. PsychometricsPsychometrika, 74(1), 121135. http://dx.doi.org/10.1007/s11336-008-9098-4

Greiner, R. (1909). Über das Fehlersystem der Kollektivmaßlehre (Of the error systemic of collectives). Journal of Mathematics and Physics / Zeitschift fur Mathematik und Physik57, 121–158, 225–260, 337–373.

Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfield, S. A. Star, & J. A. Clausen (Eds.), Measurement and prediction (pp. 60 – 90). Princeton University Press.

Harrell, F. (2001). Regression Modelling Strategies. Springer.

Harrell, F. E., Califf, R. M., Pryor, D. B., Lee, K. L., & Rosati, R. A. (1982). Evaluating the yield of medical tests. Journal of the American Medical Association247(18), 2543–2546. http://doi.org/10.1001/jama.1982.03320430047030

Heagerty, P. J., & Zheng, Y. (2005). Survival model predictive accuracy and ROC curves. Biometrics, 61(1), 92–105. https://doi.org/10.1111/j.0006-341X.2005.030814.x

Henrysson, S. (1963). Correction of item–total correlations in item analysis. PsychometricsPsychometrika, 28(2)211–218. https://doi.org/10.1007/BF02289618

Higham, P. A., & Higham, D. P. (2019). New improved gamma: Enhancing the accuracy of Goodman-Kruskal's gamma using ROC curves. Behavior Research Methods51(1), 108–125. https://doi.org/10.3758/s13428-018-1125-5

Higham, P. A., Zawadzka, K., & Hanczakowski, M. (2016). Internal mapping and its impact on measures of absolute and relative metacognitive accuracy. In J. Dunlosky & S. K. Tauber (Eds.), The Oxford handbook of metamemory. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199336746.013.15

Hryniewicz, O. (2006). Goodman-Kruskal γ measure of dependence for fuzzy ordered categorical data. Computational Statistics & Data Analysis, 51(1), 323–334. https://doi.org/10.1016/j.csda.2006.04.014 

IBM (2017). IBM SPSS Statistics 25 Algorithms. IBM.

Jonckheere, A. R. (1954). A distribution-free k–sample test against ordered alternatives. BiometricsBiometrika, 41(1–2), 133–145. http://doi.org/10.1093/biomet/41.1-2.133

KendallM. G. (1938). A new measure of rank correlation. BiometricsBiometrika30(1/2), 81–93. http://doi.org/10.2307/2332226

Kendall, M. G. (1948). Rank correlation methods (1st ed.). Charles Griffin & Co. Ltd.

Kendall, M. G. (1949). Rank and product–moment correlation. BiometricsBiometrika, 36(1/2), 177–193. https://doi.org/10.2307/2332540

Kim, J.-O. (1971). Predictive measures of ordinal association. American Journal of Sociology, 76(5), 891–907. https://doi.org/10.1086/225004

Kreiner, S., & Christensen, K. B. (2009). Item screening in graphical loglinear Rash models. PsychometricsPsychometrika, 76(2), 228–256. https://doi.org/10.1007/s11336-011-9203-y

Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks on one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. http://doi.org/10.2307/2280779

Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. PsychometricsPsychometrika2(3), 151–160. http://doi.org/10.1007/BF02288391

Kvålseth, T. O. (2017). An alternative measure of ordinal association as a value-validity correction of the Goodman–Kruskal gamma. Communications in Statistics - Theory and Methods, 46(21), 10582–10593. https://doi.org/10.1080/03610926.2016.1239114

Livingston, S. A., & Dorans, N. J. (2004). A graphical approach to item analysis (Research Report No. RR-04-10). Educational Testing Service. http://doi.org/10.1002/j.2333-8504.2004.tb01937.x

Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Addison–Wesley Publishing Company.

Love, T. E. (1997). Distractor selection ratios. PsychometricsPsychometrika, 62(1)51–62. https://doi.org/10.1007/BF02294780

Mann, H. B. (1945). Nonparametric tests against trend. EconometricsEconometrica, 13(3), 245–259. https://doi.org/10.2307/1907187

Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics18(1), 50–60. http://doi.org/10.1214/aoms/1177730491

Martin, W. S. (1973). The effects of scaling on the correlation coefficient: A test of validity. Journal of Marketing Research10(3), 316–318. http://doi.org/10.2307/3149702

Martin, W. S. (1978). Effects of scaling on the correlation coefficient: Additional considerations. Journal of Marketing Research, 15(2), 304–308. https://doi.org/10.1177/002224377801500219

Masson, M. E. J., & Rotello, C. M. (2009). Sources of bias in the Goodman–Kruskal gamma coefficient measure of association: Implications for studies of metacognitive processes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(2), 509–527. https://doi.org/10.1037/a0014876

McDonald, R. P. (1985). Factor analysis and related methods. Lawrence Erlbaum Associates.

Meade, A. W. (2010). Restriction of range. In N. J. Salkind (Ed.), Encyclopedia of research design (pp. 1278–1280). SAGE Publications, Inc. http://doi.org/10.4135/9781412961288.n309

Metsämuuronen, J. (2016). Item–total correlation as the cause for the underestimation of the alpha estimate for the reliability of the scale. GJRA - Global Journal for Research Analysis5(1), 471–477.

Metsämuuronen, J. (2017). Essentials of research methods in human sciences. SAGE Publications, Inc.

Metsämuuronen, J. (2020a). Somers’ D as an Alternative for the Item–Test and Item–Rest Correlation Coefficients in the Educational Measurement Settings. International Journal of Educational Measurement, 6(1), 207–221. https://doi.org/10.12973/ijem.6.1.207

Metsämuuronen, J. (2020b). Dimension-corrected Somers’ D for the item analysis settings. International Journal of Educational Methodology, 6(2), 297–317. https://doi.org/10.12973/ijem.6.2.297

Metsämuuronen, J. (2021). Directional nature of Goodman–Kruskal gamma and some consequences—Identity of Goodman–Kruskal gamma and Somers delta, and their connection to Jonckheere–Terpstra test statistic. ResearchGate. http://doi.org/10.13140/RG.2.2.19404.44163

Metsämuuronen, J., & Ukkola, A. (2019). Alkumittauksen menetelmällisiä ratkaisuja [Methodological solutions of zero level assessment]. Finnish Education Evaluation Centre.

Moses, T. (2017). A review of developments and applications in item analysis. In R. Bennett, & M. von Davier (Eds.), Advancing human assessmentThe methodological, psychological and policy contributions of ETS (pp. 19–46). Springer Open. http://doi.org/10.1007/978-3-319-58689-2_2

Newson, R. (2002). Parameters behind “nonparametric” statistics: Kendall’s tau, Somers’ D and median differences. The Stata Journal, 2(1), 45–64. https://doi.org/10.1177/1536867X0200200103

Newson, R. (2006). Confidence intervals for rank statistics: Somers’ D and extensions. The Stata Journal, 6(3), 309–334. https://doi.org/10.1177/1536867X0600600302

Newson, R. (2008). Identity of Somers’ D and the rank biserial correlation coefficient. https://www.rogernewsonresources.org.uk/miscdocs/ranksum1.pdf

Nielsen, J. B., Kyvsgaard, J. N., Sildorf, S. M., Kreiner, S., & Svensson, J. (2017). Item analysis using Rasch models confirms that the Danish versions of the DISABKIDS® chronic-generic and diabetes-specific modules are valid and reliable. Health Qual Life Outcomes 15(1)article 44, 1–10. https://doi.org/10.1186/s12955-017-0618-8

Nielsen, T., & Santiago, P. H. R. (2020). Using graphical loglinear Rasch models to investigate the construct validity of Perceived Stress Scale. In M. S. Khine (Ed.), Rasch Measurement: Applications in Quantitative Educational Research (pp. 261–281). Springer Nature. https://doi.org/10.1007/978-981-15-1800-3_14

Olsson, U. (1980). Measuring correlation in ordered two-way contingency tables. Journal of Marketing Research, 17(3), 391–394. https://doi.org/10.1177/002224378001700315

Pearson, K. (1896). VII. Mathematical contributions to the theory of evolution.- III. Regression, heredity and panmixia. Philosophical Transactions of the Royal Society A, 187, 253–318https://doi.org/10.1098/rsta.1896.0007

Pearson, K. (1903). I. Mathematical contributions to the theory of evolution. —XI. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society A. Mathematical, Physical and Engineering Sciences, 200(321–330), 1–66. https://doi.org/10.1098/rsta.1903.0001

Raykov, T., & Marcoulides, G. A. (2013). Meta-analysis of reliability coefficients using latent variable modeling. Structural Equation Modeling20(2), 338‒353. http://doi.org/10.1080/10705511.2013.769396

Rousson, V. (2007). The gamma coefficient revisited. Statistics & Probability Letters 77(17), 1696–1704. https://doi.org/10.1016/j.spl.2007.04.009

Sackett, P. R., Lievens, F., Berry, C. M., & Landers, R. N. (2007). A cautionary note on the effect of range restriction on predictor intercorrelations. Journal of Applied Psychology, 92(2), 538–544. http://doi.org/10.1037/0021-9010.92.2.538

Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85(1), 112–118. https://doi.org/10.1037/0021-9010.85.1.112

Sen, P. K. (1963). On the estimation of relative potency in dilution(-direct) assays by distribution-free methods. Biometrics, 19(4), 532–552. https://doi.org/10.2307%2F2527532

Shafina, A (2021). The impact of birth-order, sib-size, siblings’ sex composition on educational attainment in the Maldives. The Universal Academic Research Journal, 3(2), 87–100.

Sheskin, D. J. (2011). Handbook of parametric and nonparametric statistical procedures (5th ed.). Chapman & Hall/CRC.

Siegel, S., & Castellan, N. J., Jr. (1988). Nonparametric statistics for the behavioural sciences (2nd ed.). McGraw-Hill.

Sirkin, M. R. (2006). Statistics of the social science (3rd ed.). SAGE Publications, Inc.

Somers, R. H. (1962). A new asymmetric measure of association for ordinal variables. American Sociological Review, 27(6), 799–811. http://doi.org/10.2307/2090408

Somers, R. H. (1980). Simple approximations to null sampling variances. Goodman and Kruskal’s gamma, Kendall’s tau and Somers dyxSociological Methods & Research, 9(1), 115–126. https://doi.org/10.1177/004912418000900107

Terpstra, T. J. (1952). The asymptotic normality and consistency of Kendall’s test against trend, when ties are present in one ranking. Investigations of the mathematics/ Indagationes Mathematicae, 14(3), 327–333. http://doi.org/10.1016/S1385-7258(52)50043-X

Theil, H. (1950). A rank-invariant method of linear and polynomial regression analysis I, II, III. In Proceedings of the Section of Sciences - Koninklijke Nederlandsche Akademie van Wetenschappen [Royal Netherlands Academy of Sciences] (Series A. Mathematical Sciences, pp. 386–392, 521–525, 1397–1412). North-Holland.

Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best alternatives to Cronbach's alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7, 1-8. https://doi.org/10.3389/fpsyg.2016.00769

Van der Ark, L. A., & Van Aert, R. C. M. (2015). Comparing confidence intervals for Goodman and Kruskal's gamma coefficient. Journal of Statistical Computation and Simulation, 85(12), 2491–2505. https://doi.org/10.1080/00949655.2014.932791 

Wholey, J., S., Hatry, H., P., & Newcomer, K. E. (Eds.) (2015). Handbook of practical program evaluation (4th ed.). Jossey-Bass.

Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin1(6), 80–83. http://doi.org/10.2307/3001968

Wilson, T. P. (1974). Measures of association for bivariate ordinal hypotheses. In H. M. Blalock (Ed.), Measurement in the social sciences (pp. 327–342). Aldine.

Zaionts, C. (2020). Polychoric correlation using solver. Real Statistics Using Excel. http://www.real-statistics.com/correlation/polychoric-correlation/polychoric-correlation-using-solver/

...