Comparing group means with the total mean in random samples, surveys, and large-scale assessments: A tutorial and software illustration.
Large-scale assessment
Linear regression
Mean comparison
Survey
Weighted effect coding
Journal
Behavior research methods
ISSN: 1554-3528
Titre abrégé: Behav Res Methods
Pays: United States
ID NLM: 101244316
Informations de publication
Date de publication:
06 2022
06 2022
Historique:
accepted:
21
01
2021
pubmed:
26
9
2021
medline:
9
6
2022
entrez:
25
9
2021
Statut:
ppublish
Résumé
In many disciplines of the social sciences, comparisons between a group mean and the total mean is a common but also challenging task. As one solution to this statistical testing problem, we propose using linear regression with weighted effect coding. For random samples, this procedure is straightforward and easy to implement by means of standard statistical software. However, for complex or clustered samples with imputed or weighted data, which are common in survey analyses, there is a lack of easy-to-use software solutions. In this paper, we discuss scenarios that are commonly encountered in the social sciences such as heterogeneous variances, weighted samples, and clustered samples, and we describe how group means can be compared to the total mean in these situations. We introduce the R package eatRep, which is a front end that makes the presented methods easily accessible for researchers. Two empirical examples, one using survey data (MIDUS 1) and the other using large-scale assessment data (PISA 2015), are given for illustration. Annotated R code to run group to total mean comparisons is provided.
Identifiants
pubmed: 34561822
doi: 10.3758/s13428-021-01553-1
pii: 10.3758/s13428-021-01553-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1051-1062Informations de copyright
© 2021. The Psychonomic Society, Inc.
Références
Beaujean, A. A. (2014). Latent variable modeling using R: A Step-by-Step guide. Routledge
Bell, R. M., & McCaffrey, D. F. (2002). Bias Reduction in Standard Errors for Linear Regression with Multi-Stage Samples. Survey Methodology, 28(2), 169-182
Blair, G., Cooper, J., Coppock, A., Humphreys, M., & Sonnet, L. (2020). estimatr: Fast Estimators for Design-Based Inference (Version R package version 0.22.0). Retrieved from https://cran.r-project.org/web/packages/estimatr/index.html
Brim, O. G., Baltes, P. B., Bumpass, L. L., Cleary, P. D., Featherman, D. L., Hazzard, W. R., ... Shweder, R. A. (2019). Midlife in the United States (MIDUS 1), 1995-1996 [Data file, documentation, and code book]. Inter-university Consortium for Political and Social Research
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3), 414-427
Davison, A. C., & Hinkley, D. V. (1997). Bootstrap Methods and Their Applications. Cambridge University Press
Efron, B., & Tibshirani, R. (1986). Bootstrap Methods for Standard Errors, Confidence Intervals, and other Measures of Statistical Accuracy. Statistical Science, 1(1), 54-77
Feng, Z., McLerran, D., & Grizzle, J. (1996). A comparison of statistical Methods for clustered data analysis with Gaussian error. Statistics in Medicine, 15, 1793-1806
Foy, P., Galia, J., & Li, I. (2008). Scaling the data from the TIMSS 2007 mathematics and science asssessment. In J. F. Olson, M. O. Martin, & I. V. S. Mullis (Eds.), TIMSS 2007 Technical Report (pp. 225-280). TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College
Freedman, D. A. (2006). On the So-Called "Huber Sandwich Estimator" and "Robust Standard Errors". The American Statistician, 60(4), 299-302
Gonzalez, E. (2014). Calculating standard errors of sample statistics when using international large-scale assessment data. In R. Striethold, W. Bos, J.-E. Gustafsson, & M. Rosén (Eds.), Educational policy evaluation through international comparative assessments. Waxmann
Harden, J. J. (2011). A Bootstrap Method for Conducting Statistical Inference with Clustered Data. State Politics & Policy Quaterly, 11(2), 223-246
Harel, O., & Schafer, J. L. (2003). Multiple imputation in two stages. Paper presented at the Proceedings of Federal Committee on Statistical Methodology Research Conference, Washington DC
Judkins, D. R. (1990). Fay’s Method for Variance Estimation. Journal of Statistics, 6, 223-229
Kalpic, D., Hlupic, N., & Lovric, M. (2011). Student’s t-tests. In M. Lovric (Ed.), International encyclopedia of statistical science (pp. 1559-1563). Springer
Krewski, D., & Rao, J. N. K. (1981). Inference From Stratified Samples: Properties of the Linearization, Jackknife and Balanced Repeated Replication Method. Annals of Statistics, 9, 1010-1019
Little, R. J. A., & Rubin, D. B. (1987). Statistical Analyses with Missing Data. Wiley
Lumley, T. (2004). Analysis of complex survey samples. Journal of Statistical Software, 9(1), 1-19
Lumley, T. (2019). survey: analysis of complex survey samples (Version 3.35-1)
MacKinnon, J., & White, H. (1985). Some Heteroskedasticity-Consistent Covariance Matrix Estimators with Improved Finite Sample Properties. Journal of Econometrics, 29(3), 305-325
Magnussen, S., McRoberts, R. E., & Tomppo, E. O. (2010). A resampling variance estimator for the k nearest neighbours technique. Canadian Journal of Forest Research, 40(4), 648-658
Mayer, A., Dietzfelbinger, L., Rosseel, Y., & Steyer, R. (2016). The EffectLiteR approach for analyzing average and conditional effects. Multivariate Behavioral Research, 51, 374-391
Mayer, A., & Thoemmes, F. (2019). Analysis of Variance Models with Stochastic Group Weights. Multivariate Behavioral Research, 54(4), 542-554
McIntosh, A. (2016). The Jackknife Estimation Method. Retrieved from https://arxiv.org/abs/1606.00497v1
Mislevy, R. J., Beaton, A. E., Kaplan, B., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133-161. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=1992-41607-001&site=ehost-live
Mullis, I. V. S., Martin, M. O., Foy, P., Olson, J. F., Preuschoff, C., Erberber, E., ... Galia, J. (2008). TIMSS 2007 International Mathematics Report: Findings from IEA’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades. TIMSS & PIRLS International Study Center
Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus User's Guide (8th ed.). Muthén & Muthén
OECD. (2005). PISA 2003 Data Analysis Manual: SAS Users. OECD
OECD. (2016). PISA 2015 Results (Volume I). Excellence and equity in education. OECD Publishing
OECD. (2017). PISA 2015 Technical Report. OECD Publishing
R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.1). R Foundation for Statistical Computing
Rabe-Hesketh, S., & Skrondal, A. (2006). Multilevel modelling of complex survey data. Journal of Royal Statistical Society, 169(4), 805-827
Rao, J. N. K., & Wu, C. F. J. (1985). Inference From Stratified Samples: Second-Order Analysis of Three Methods for Nonlinear Statistics. Journal of the American Statistical Association, 80(391), 620-630
Reiter, J. P., & Raghunathan, T. E. (2007). The multiple Adaptions of Multiple Imputation. Journal of the American Statistical Association, 102, 1462-1471
Robitzsch, A., & Oberwimmer, K. (2019). BIFIEsurvey: Tools for survey statistics in educational assessment (Version 3.3-12)
Rogers, W. H. (1993). Regression standard errors in clustered samples. Stata Technical Bulletin, 13, 19-23. Retrieved from https://www.stata.com/support/faqs/statistics/stb13_rogers.pdf
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1-36
Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. Wiley
Rubin, D. B. (2003). Nested multiple imputation of NMES via partially incompatible MCMC. Statistica Neerlandica, 57(1), 3-18
Rust, K. (2014). Sampling, weighting, and variance estimation. In L. Rutkowski, M. Von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment. CRC Press
Rust, K., & Rao, J. N. K. (1996). Variance estimation for complex surveys using replication techniques. Statistical Methods in Medical Research, 5, 283-310
Rutkowski, L., Gonzalez, E., Joncas, M., & Von Davier, M. (2010). International Large-Scale Assessment Data: Issues in Secondary Analysis and Reporting. Educational Researcher, 39(2), 142-151
Salkind, N. (2010). Encyclopedia of research design. Sage
SAS Institute Inc. (2018). SAS/STAT 15.1 User’s Guide. SAS Institute Inc
Schofield, W. (2006). Survey Sampling. In R. Sapsford & V. Jupp (Eds.), Data Collection and Analysis (pp. 26-56). Sage
Searle, S. R. (1971). Linear Models. Wiley
Sherman, M., & le Cessie, S. (1997). A comparison between bootstrap methods and generalized estimating equations for correlated outcomes in generalized linear models. Communications in Statistics - Simulation and Computation, 26(3), 901-925
Skinner, C. J., & Wakefiel, J. (2017). Introduction to the design and analysis of complex survey data. Statistical Science, 32(2), 165-175
Smyth, G. K. (2002). An efficient algorithm for REML in heteroscedastic regression. Journal of Computational and Graphical Statistics, 11, 836-847
Sweeney, R. E., & Ulveling, E. F. (1972). A Transformation for Simplifying the Interpretation of Coefficients of Binary Variablesin Regression Analysis. The American Statistician, 26(5), 30-32
te Grotenhuis, M., Pelzer, B., Eisinga, R., Nieuwenhuis, R., Schmidt-Catran, A., & Konig, R. (2017). When size matters: advantages of weighted effect coding in observational studies. International Journal of Public Health, 62, 163-167
van Buuren, S. (2007). Multiple imputation of discrete and continuous data by fully conditional specification. Statistical Methods in Medical Research, 16, 219-242
van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1-67
von Davier, M., Gonzalez, E., & Mislevy, R. J. (2009). What are plausible values and why are they useful? IERI Monograph Series: Issues and Methodologies in Large Scale Assessments, 2, 9-36
Weirich, S., Haag, N., Hecht, M., Böhme, K., Siegle, T., & Lüdtke, O. (2014). Nested multiple imputation in large-scale assessments. Large-scale Assessments in Education, 2(9), 1-18
Weirich, S., Hecht, M., & Becker, B. (2020). Educational Assessment Tools for Replication Methods (Version R package version 0.13.4). Retrieved from https://cran.r-project.org/web//packages/eatRep/index.html
Westat. (2000). WesVar. Westat
Westfall, P. H. (2011). On Using the Bootstrap for Multiple Comparisons. Journal of Biopharmaceutical Statistics, 21(6), 1187-1205
White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817-838.
Wickham, H. (2007). Reshaping Data with the reshape Package. Journal of Statistical Software, 21(12), 1-20
Wickham, H., & Henry, L. (2020). tidyr: Tidy Messy Data (Version R package version 1.1.0). Retrieved from https://cran.r-project.org/web/packages/tidyr/index.html
Wolter, K. M. (1985). Introduction to variance estimation. Springer
Zeileis, A. (2004). Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 11, 1-17