Authors retain the copyright without restrictions for their published content in this journal. HSSR is a SHERPA ROMEO Green Journal.
Publishing License
This is an open-access article distributed under the terms of
PROPER SAMPLE SIZES FOR ENGLISH LANGUAGE TESTING: A SIMPLE STATISTICAL ANALYSIS
Corresponding Author(s) : Roderick Julian Robillos
Humanities & Social Sciences Reviews,
Vol. 8 No. 4 (2020): July
Abstract
Purpose of study: Small sample size is the most common limitation which restricts the generalization of research results, and this is true to many fields, including language testing. The current study is sought to show the predictive power of sample sizes over the population mean to decide what sample minimum size can be considered as a proper sample size for a language test.
Methodology: The data for this quantitative research was 5,250 paper-based TOEFL test scores considered as the population, which includes listening, structure, and reading tests, and it is the most familiar standardized test among EFL researchers. Due to its objective nature, it leaves little chance for bias scores. The score ranged between 30.7% of 417 in the TOEFL scale and 95.7% or 653. Standard error was used as the parameter in deciding the proper sample size. It was the cut-off point when the parameter did not show any obvious change when the sample size was added. We used hierarchical agglomerative clustering with three clusters, determined using 30 indices through the majority rule, in finding out the cut-off point.
Main Findings: It was found that the cut-off point is at the sample size of 52 with the range between 46 and 59. Therefore, it can be concluded that the minimum proper sample size for a research study involving a language test is n = 46.
Application of this study: The results of this study apply to the area of English language teaching and testing. However, it does not rule out the possibility that the study result applies to tests in other languages.
Novelty/Originality of this study: The result of this study should be treated as statistical evidence of the proper sample size to avoid inaccurate or conflicting research results in language teaching where a test is used for analysis.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- Adams, K. A., & Lawrence, E. K. (2015). Research methods, statistics, and applications. Sage Publications.
- Agresti, A. (2019). An introduction to categorical analysis (3rd ed.). John Wiley & Sons, Inc.
- Ah-Pine, J. (2018). An efficient and effective generic agglomerative hierarchical clustering approach. Journal of Machine Learning Research, 19, 1–43.
- Atai, M. R., & Nazari, O. (2011). Exploring reading comprehension needs of Iranian EAP students of health information management (HIM): A triangulated approach. System, 39(1), 30–43. https://doi.org/10.1016/j.system.2011.01.015 DOI: https://doi.org/10.1016/j.system.2011.01.015
- Baby, P., & Sasirekha, K. (2013). Agglomerative hierarchical clustering algorithm- A review. International Journal of Scientific and Research Publications, 3(3), 1–3.
- Baese-Berk, M. M., & Morrill, T. H. (2015). Speaking rate consistency in native and non-native speakers of English. The Journal of the Acoustical Society of America, 138(3), 223228. https://doi.org/10.1121/1.4929622 DOI: https://doi.org/10.1121/1.4929622
- Bhalerao, S., & Kadam, P. (2010). Sample size calculation. International Journal of Ayurveda Research, 1(1), 55–57. https://doi.org/10.4103/0974-7788.59946 DOI: https://doi.org/10.4103/0974-7788.59946
- Camilli, G., & Hopkins, K. D. (1978). Applicability of Chi-square to 2 X 2 contingency tables with small expected cell frequencies. Psychological Bulletin, 85(1), 163–167. DOI: https://doi.org/10.1037/0033-2909.85.1.163
- Charrad, M., Ghazzali, N., Boiteau, V., & Niknafs, A. (2014). NbClust: An R package for determining the relevant number of clusters in a data set. Journal of Statistical Software, 61(6), 1–36. DOI: https://doi.org/10.18637/jss.v061.i06
- Coolican, H. (2014). Research methods and statistics in psychology (6th Ed.). Psychology Press. https://doi.org/doi:10.4324/9780203769669 DOI: https://doi.org/10.4324/9780203769669
- Coxhead, A. (2017). Dealing with low response rates in quantitative studies. In J. McKinley & H. Rose (Eds.), Doing research in applied linguistics: realities, dilemmas and solutions (pp. 81–90). Routledge Taylor & Francis Group. https://doi.org/10.4324/9781315389608-8 DOI: https://doi.org/10.4324/9781315389608-8
- Doryei, Z. (2007). Research Methods in Applied Linguistics: Quantitative, qualitative, and mixed methodologies. Oxford University Press. https://doi.org/10.1017/S0272263110000094 DOI: https://doi.org/10.1017/S0272263110000094
- Double, K. S., McGrane, J. A., & Hopfenbeck, T. N. (2019). The Impact of peer assessment on academic performance: A meta-analysis of control group studies. Educational Psychology Review, 31(4), 967–989. https://doi.org/10.1007/s10648-019-09510-3 DOI: https://doi.org/10.1007/s10648-019-09510-3
- Elfiondri, Kasim, U., Mustafa, F., Putra, T. M. (2020). Reading comprehension in the TOEFL PBT: Which sub-skill deserves more intensive training? TESOL International Journal, 15(1), 53–64.
- Faber, J., & Fonseca, L. M. (2014). How sample size influences research outcomes. Dental Press Journal of Orthodontics, 19(4), 27–29. https://doi.org/10.1590/2176-9451.19.4.027-029.EBO DOI: https://doi.org/10.1590/2176-9451.19.4.027-029.ebo
- Fadiana, D., Bahri Ys, S., & Inayah, N. (2020). Teaching vocabulary by using total physical response. Research in English and Education (READ), 5(1), 1–6.
- Fageeh, A. I. (2014). The use of journal writing and reading comprehension texts during pre-writing in developing EFL students’ academic writing. Studies in Literature and Language, 9(3), 1–18.
- Farvardin, M. T. (2019). Effects of spacing techniques on EFL learners’ recognition and production of lexical collocations. Indonesian Journal of Applied Linguistics, 9(2), 395–403. https://doi.org/10.17509/ijal.v9i2.20237 DOI: https://doi.org/10.17509/ijal.v9i2.20237
- French, B. F., Immekus, J. C., & Yen, H.-J. (2013). Logistic regression. In T. Teo (Ed.), Handbook of quantitative methods for educational research (pp. 145–165). Sense Publishers. DOI: https://doi.org/10.1007/978-94-6209-404-8_7
- Goh, C. C. M., & Foong, K. P. (1997). Chinese ESL students’ learning strategies: A look at frequency, proficiency, and gender. Hong Kong Journal of Applied Linguistics, 2(1), 39–53. http://eric.ed.gov/?id=EJ597324
- Gonulal, T. (2016). Statistical literacy among second language acquisition graduate students. Michigan State University.
- Gravetter, F., & Forzano, L.-A. (2012). Research Methods for The Behavioral Sciences (4th ed.). Wadsworth, Cengage Learning.
- Harfitt, G. J. (2015). Class size reduction: Key insights from secondary school classrooms. Springer. https://doi.org/10.1007/978-981-287-564-8 DOI: https://doi.org/10.1007/978-981-287-564-8
- Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. American Statistician, 69(4), 371–386. https://doi.org/10.1080/00031305.2015.1089789 DOI: https://doi.org/10.1080/00031305.2015.1089789
- Kasim, U., Muslem, A., & Mustafa, F. (2020). Empirical evidence on the effectiveness of Learning by Teaching technique among English as a foreign language university students. Journal of Language and Education, 6(4). DOI: https://doi.org/10.17323/jle.2020.10846
- Khany, R., & Tazik, K. (2019). Levels of statistical use in applied linguistics research articles: From 1986 to 2015. Journal of Quantitative Linguistics, 26(1), 48–65. https://doi.org/10.1080/09296174.2017.1421498 DOI: https://doi.org/10.1080/09296174.2017.1421498
- Kothari, C. R. (2004). Research methodology: Methods and techniques (2nd Ed). New Age International (P) Ltd.
- Latifi, M., Mobalegh, A., & Mohammadi, E. (2011). Movie subtitles and the improvement of listening comprehension ability: Does it help? The Journal of Language Teaching and Learning, 1(2), 18–29.
- Lock, R. H., Lock, P. F., Morgan, K. L., Lock, E. F., & Lock, D. F. (2017). Statistics: Unlocking the power of data (2nd ed.). John Wiley & Sons, Inc.
- Mendenhall, W. I., Beaver, R. J., & Beaver, B. M. (2013). Introduction to Probaility and Statistics. https://doi.org/10.1017/CBO9781107415324.004 DOI: https://doi.org/10.1017/CBO9781107415324.004
- Mustafa, F., & Anwar, S. (2018). Distinguishing TOEFL score: What is the lowest score considered a TOEFL score? Pertanika Journal of Social Sciences and Humanities, 26(3), 1995–2008.
- Navarro, D. (2016). Learning statistics with R: A tutorial for psychology students and other beginners. University of New South Wales.
- Neuman, W. L. (2014). Social research methods: Qualitative and quantitative approaches (7th ed.). Pearson Education Limited.
- Nikitina, L., Paidi, R., & Furuoka, F. (2019). Using bootstrapped quantile regression analysis for small sample research in applied linguistics: Some methodological considerations. PLoS ONE, 14(1), 1–19. https://doi.org/10.1371/journal.pone.0210668 DOI: https://doi.org/10.1371/journal.pone.0210668
- Nirwan, N. (2020). Using KWL (know-want to know-learned) strategy in improving students’ reading comprehension. English Education Journal, 11(2), 199–214.
- Peacock, M. (2002). Communicative moves in the discussion section of research articles. System, 30, 479–497. DOI: https://doi.org/10.1016/S0346-251X(02)00050-7
- Perakyla, A. (1997). Reliability and validity in research based on naturally occurring social interaction. In D. Silverman (Ed.), Qualitative research: Theory, method and practice (2nd Ed., pp. 283–304). Sage Productions.
- Privitera, G. J. (2018). Statistics for the behavioral sciences (3rd Ed). Sage Production.
- Razaghi, M., Bagheri, M. S., & Yamini, M. (2019). The impact of cognitive scaffolding on Iranian EFL learners’ speaking skill. International Journal of Instruction, 12(4), 95–112. https://doi.org/10.29333/iji.2019.1247a DOI: https://doi.org/10.29333/iji.2019.1247a
- Ruiying, Y., & Allison, D. (2003). Research articles in applied linguistics: Moving from results to conclusions. English for Specific Purposes, 22(4), 365–385. https://doi.org/10.1016/S0889-4906(02)00026-1 DOI: https://doi.org/10.1016/S0889-4906(02)00026-1
- Sadia, F., & Hossain, S. S. (2014). Contrast of Bayesian and classical sample size determination. Journal of Modern Applied Statistical Methods, 13(2), 420–431. https://doi.org/10.22237/jmasm/1414815720 DOI: https://doi.org/10.22237/jmasm/1414815720
- Setiawan, M. R., & Wiedarti, P. (2020). The effectiveness of Quizlet application towards students’ motivation in learning vocabulary. Studies in English Language and Education, 7(1), 83–95. https://doi.org/10.24815/siele.v7i1.15359 DOI: https://doi.org/10.24815/siele.v7i1.15359
- Shieh, W., & Freiermuth, M. R. (2010). Using the DASH Method to Measure Reading Comprehension. TESOL Quarterly, 44(1), 110–128. https://doi.org/10.5054/tq.2010.217676 DOI: https://doi.org/10.5054/tq.2010.217676
- Slim, H., & Hafedh, M. (2019). Social media impact on language learning for specific purposes: A study in English for business administration. Teaching English with Technology, 19(1), 56–71.
- Stangor, C. (2011). Research methods for the behavioral sciences (4th ed.). Wadsworth, Cengage Learning.
- Tuckman, B. W., & Harper, B. E. (2012). Conducting educational research (6th Ed). Rowman & Littlefield Publishers, Inc.
- VanVoorhis, C. R. W., & Morgan, B. L. (2007). Understanding power and rules of thumb for determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(2), 43–50. DOI: https://doi.org/10.20982/tqmp.03.2.p043
- Vaux, A., & Briggs, C. S. (2006). Conducting mail and internet surveys. In F. T. L. Leong & J. T. Austin (Eds.), The Psychology Research Handbook: A Guide for Graduate Students and Research Assistants (pp. 186–209). SAGE Publications, Inc. https://doi.org/10.4135/9781412976626.n13 DOI: https://doi.org/10.4135/9781412976626.n13
- Wei, R., Hu, Y., & Xiong, J. (2019). Effect size reporting practices in applied linguistics research: A study of one major journal. SAGE Open, 9(2). https://doi.org/10.1177/2158244019850035 DOI: https://doi.org/10.1177/2158244019850035
- Wu, M. M. (2007). The relationships between the use of metacognitive language-learning strategies and language-learning motivation among Chinese-speaking ESL learners at a vocational education institute in Hong Kong. Asian EFL Journal, 9(3), 93–117.
References
Adams, K. A., & Lawrence, E. K. (2015). Research methods, statistics, and applications. Sage Publications.
Agresti, A. (2019). An introduction to categorical analysis (3rd ed.). John Wiley & Sons, Inc.
Ah-Pine, J. (2018). An efficient and effective generic agglomerative hierarchical clustering approach. Journal of Machine Learning Research, 19, 1–43.
Atai, M. R., & Nazari, O. (2011). Exploring reading comprehension needs of Iranian EAP students of health information management (HIM): A triangulated approach. System, 39(1), 30–43. https://doi.org/10.1016/j.system.2011.01.015 DOI: https://doi.org/10.1016/j.system.2011.01.015
Baby, P., & Sasirekha, K. (2013). Agglomerative hierarchical clustering algorithm- A review. International Journal of Scientific and Research Publications, 3(3), 1–3.
Baese-Berk, M. M., & Morrill, T. H. (2015). Speaking rate consistency in native and non-native speakers of English. The Journal of the Acoustical Society of America, 138(3), 223228. https://doi.org/10.1121/1.4929622 DOI: https://doi.org/10.1121/1.4929622
Bhalerao, S., & Kadam, P. (2010). Sample size calculation. International Journal of Ayurveda Research, 1(1), 55–57. https://doi.org/10.4103/0974-7788.59946 DOI: https://doi.org/10.4103/0974-7788.59946
Camilli, G., & Hopkins, K. D. (1978). Applicability of Chi-square to 2 X 2 contingency tables with small expected cell frequencies. Psychological Bulletin, 85(1), 163–167. DOI: https://doi.org/10.1037/0033-2909.85.1.163
Charrad, M., Ghazzali, N., Boiteau, V., & Niknafs, A. (2014). NbClust: An R package for determining the relevant number of clusters in a data set. Journal of Statistical Software, 61(6), 1–36. DOI: https://doi.org/10.18637/jss.v061.i06
Coolican, H. (2014). Research methods and statistics in psychology (6th Ed.). Psychology Press. https://doi.org/doi:10.4324/9780203769669 DOI: https://doi.org/10.4324/9780203769669
Coxhead, A. (2017). Dealing with low response rates in quantitative studies. In J. McKinley & H. Rose (Eds.), Doing research in applied linguistics: realities, dilemmas and solutions (pp. 81–90). Routledge Taylor & Francis Group. https://doi.org/10.4324/9781315389608-8 DOI: https://doi.org/10.4324/9781315389608-8
Doryei, Z. (2007). Research Methods in Applied Linguistics: Quantitative, qualitative, and mixed methodologies. Oxford University Press. https://doi.org/10.1017/S0272263110000094 DOI: https://doi.org/10.1017/S0272263110000094
Double, K. S., McGrane, J. A., & Hopfenbeck, T. N. (2019). The Impact of peer assessment on academic performance: A meta-analysis of control group studies. Educational Psychology Review, 31(4), 967–989. https://doi.org/10.1007/s10648-019-09510-3 DOI: https://doi.org/10.1007/s10648-019-09510-3
Elfiondri, Kasim, U., Mustafa, F., Putra, T. M. (2020). Reading comprehension in the TOEFL PBT: Which sub-skill deserves more intensive training? TESOL International Journal, 15(1), 53–64.
Faber, J., & Fonseca, L. M. (2014). How sample size influences research outcomes. Dental Press Journal of Orthodontics, 19(4), 27–29. https://doi.org/10.1590/2176-9451.19.4.027-029.EBO DOI: https://doi.org/10.1590/2176-9451.19.4.027-029.ebo
Fadiana, D., Bahri Ys, S., & Inayah, N. (2020). Teaching vocabulary by using total physical response. Research in English and Education (READ), 5(1), 1–6.
Fageeh, A. I. (2014). The use of journal writing and reading comprehension texts during pre-writing in developing EFL students’ academic writing. Studies in Literature and Language, 9(3), 1–18.
Farvardin, M. T. (2019). Effects of spacing techniques on EFL learners’ recognition and production of lexical collocations. Indonesian Journal of Applied Linguistics, 9(2), 395–403. https://doi.org/10.17509/ijal.v9i2.20237 DOI: https://doi.org/10.17509/ijal.v9i2.20237
French, B. F., Immekus, J. C., & Yen, H.-J. (2013). Logistic regression. In T. Teo (Ed.), Handbook of quantitative methods for educational research (pp. 145–165). Sense Publishers. DOI: https://doi.org/10.1007/978-94-6209-404-8_7
Goh, C. C. M., & Foong, K. P. (1997). Chinese ESL students’ learning strategies: A look at frequency, proficiency, and gender. Hong Kong Journal of Applied Linguistics, 2(1), 39–53. http://eric.ed.gov/?id=EJ597324
Gonulal, T. (2016). Statistical literacy among second language acquisition graduate students. Michigan State University.
Gravetter, F., & Forzano, L.-A. (2012). Research Methods for The Behavioral Sciences (4th ed.). Wadsworth, Cengage Learning.
Harfitt, G. J. (2015). Class size reduction: Key insights from secondary school classrooms. Springer. https://doi.org/10.1007/978-981-287-564-8 DOI: https://doi.org/10.1007/978-981-287-564-8
Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. American Statistician, 69(4), 371–386. https://doi.org/10.1080/00031305.2015.1089789 DOI: https://doi.org/10.1080/00031305.2015.1089789
Kasim, U., Muslem, A., & Mustafa, F. (2020). Empirical evidence on the effectiveness of Learning by Teaching technique among English as a foreign language university students. Journal of Language and Education, 6(4). DOI: https://doi.org/10.17323/jle.2020.10846
Khany, R., & Tazik, K. (2019). Levels of statistical use in applied linguistics research articles: From 1986 to 2015. Journal of Quantitative Linguistics, 26(1), 48–65. https://doi.org/10.1080/09296174.2017.1421498 DOI: https://doi.org/10.1080/09296174.2017.1421498
Kothari, C. R. (2004). Research methodology: Methods and techniques (2nd Ed). New Age International (P) Ltd.
Latifi, M., Mobalegh, A., & Mohammadi, E. (2011). Movie subtitles and the improvement of listening comprehension ability: Does it help? The Journal of Language Teaching and Learning, 1(2), 18–29.
Lock, R. H., Lock, P. F., Morgan, K. L., Lock, E. F., & Lock, D. F. (2017). Statistics: Unlocking the power of data (2nd ed.). John Wiley & Sons, Inc.
Mendenhall, W. I., Beaver, R. J., & Beaver, B. M. (2013). Introduction to Probaility and Statistics. https://doi.org/10.1017/CBO9781107415324.004 DOI: https://doi.org/10.1017/CBO9781107415324.004
Mustafa, F., & Anwar, S. (2018). Distinguishing TOEFL score: What is the lowest score considered a TOEFL score? Pertanika Journal of Social Sciences and Humanities, 26(3), 1995–2008.
Navarro, D. (2016). Learning statistics with R: A tutorial for psychology students and other beginners. University of New South Wales.
Neuman, W. L. (2014). Social research methods: Qualitative and quantitative approaches (7th ed.). Pearson Education Limited.
Nikitina, L., Paidi, R., & Furuoka, F. (2019). Using bootstrapped quantile regression analysis for small sample research in applied linguistics: Some methodological considerations. PLoS ONE, 14(1), 1–19. https://doi.org/10.1371/journal.pone.0210668 DOI: https://doi.org/10.1371/journal.pone.0210668
Nirwan, N. (2020). Using KWL (know-want to know-learned) strategy in improving students’ reading comprehension. English Education Journal, 11(2), 199–214.
Peacock, M. (2002). Communicative moves in the discussion section of research articles. System, 30, 479–497. DOI: https://doi.org/10.1016/S0346-251X(02)00050-7
Perakyla, A. (1997). Reliability and validity in research based on naturally occurring social interaction. In D. Silverman (Ed.), Qualitative research: Theory, method and practice (2nd Ed., pp. 283–304). Sage Productions.
Privitera, G. J. (2018). Statistics for the behavioral sciences (3rd Ed). Sage Production.
Razaghi, M., Bagheri, M. S., & Yamini, M. (2019). The impact of cognitive scaffolding on Iranian EFL learners’ speaking skill. International Journal of Instruction, 12(4), 95–112. https://doi.org/10.29333/iji.2019.1247a DOI: https://doi.org/10.29333/iji.2019.1247a
Ruiying, Y., & Allison, D. (2003). Research articles in applied linguistics: Moving from results to conclusions. English for Specific Purposes, 22(4), 365–385. https://doi.org/10.1016/S0889-4906(02)00026-1 DOI: https://doi.org/10.1016/S0889-4906(02)00026-1
Sadia, F., & Hossain, S. S. (2014). Contrast of Bayesian and classical sample size determination. Journal of Modern Applied Statistical Methods, 13(2), 420–431. https://doi.org/10.22237/jmasm/1414815720 DOI: https://doi.org/10.22237/jmasm/1414815720
Setiawan, M. R., & Wiedarti, P. (2020). The effectiveness of Quizlet application towards students’ motivation in learning vocabulary. Studies in English Language and Education, 7(1), 83–95. https://doi.org/10.24815/siele.v7i1.15359 DOI: https://doi.org/10.24815/siele.v7i1.15359
Shieh, W., & Freiermuth, M. R. (2010). Using the DASH Method to Measure Reading Comprehension. TESOL Quarterly, 44(1), 110–128. https://doi.org/10.5054/tq.2010.217676 DOI: https://doi.org/10.5054/tq.2010.217676
Slim, H., & Hafedh, M. (2019). Social media impact on language learning for specific purposes: A study in English for business administration. Teaching English with Technology, 19(1), 56–71.
Stangor, C. (2011). Research methods for the behavioral sciences (4th ed.). Wadsworth, Cengage Learning.
Tuckman, B. W., & Harper, B. E. (2012). Conducting educational research (6th Ed). Rowman & Littlefield Publishers, Inc.
VanVoorhis, C. R. W., & Morgan, B. L. (2007). Understanding power and rules of thumb for determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(2), 43–50. DOI: https://doi.org/10.20982/tqmp.03.2.p043
Vaux, A., & Briggs, C. S. (2006). Conducting mail and internet surveys. In F. T. L. Leong & J. T. Austin (Eds.), The Psychology Research Handbook: A Guide for Graduate Students and Research Assistants (pp. 186–209). SAGE Publications, Inc. https://doi.org/10.4135/9781412976626.n13 DOI: https://doi.org/10.4135/9781412976626.n13
Wei, R., Hu, Y., & Xiong, J. (2019). Effect size reporting practices in applied linguistics research: A study of one major journal. SAGE Open, 9(2). https://doi.org/10.1177/2158244019850035 DOI: https://doi.org/10.1177/2158244019850035
Wu, M. M. (2007). The relationships between the use of metacognitive language-learning strategies and language-learning motivation among Chinese-speaking ESL learners at a vocational education institute in Hong Kong. Asian EFL Journal, 9(3), 93–117.