Authors retain the copyright without restrictions for their published content in this journal. HSSR is a SHERPA ROMEO Green Journal.
Publishing License
This is an open-access article distributed under the terms of
THE OPTIMAL NUMBER OF OPTIONS USED IN MULTIPLE-CHOICE TEST FORMAT FOR NATIONAL EXAMINATIONS IN INDONESIA
Corresponding Author(s) : Herland Franley Manalu
Humanities & Social Sciences Reviews,
Vol. 8 No. 2 (2020): March
Abstract
Purpose of the study: The present research purports to find out the most favourable number of options used in the Multiple Choice (MC) format for Senior High Schools in Indonesian National Examinations (UN) and suggest the testing division within the Indonesia ministry of education to consider the result of the study
Methodology: Two English tests using MC questions with five options, consisting of 50 questions per test format, were used as the elicitation devices for this research. One of these English tests was rewritten to create four options by deleting the non-functioning distractors in each question. Both tests with different MC test formats were administered to 2 groups within two state Senior High Schools (SMA). The Classical Method and Rasch Analysis were utilized to compare item facility, item discrimination, distractor measure correlation, and reliabilities across the two MC test formats. To corroborate the findings of the study, questionnaires were randomly distributed to 120 SMA students and 15 SMA English teachers.
Main Findings: The findings suggested that four options were more difficult than five options. There was significant change observed in Item Facility (p<0.05) and Item Discrimination across the two MC test formats. Based on the questionnaire data analysis, the four-option is the more optimal and preferable format to be used in the National Examinations.
Applications of this study: Regarding the practicality issues like saving time and money for implementing the tests, minimizing the amount of time and effort needed for test-makers to create the tests, and also reducing the risk of providing implausible distractors for developing MC tests, this study concluded that the four-option MC format is more optimal to be used for Indonesian National Examinations in Senior High School.
Novelty/Originality of this study: The testing division within the Indonesia Ministry of Education uses different numbers of MC options in Elementary, Junior High, and Senior High Schools examinations. The Ministry has predetermined three options for elementary schools, four options for junior high schools, and five options for senior high school students. The decision made the researchers of the present study eager to find out whether the use of MC format with reduced options in the UN SMA will produce the same results or maybe increase or reduce the efficiency, effectiveness, reliability, and practicality of the test administration. This is the first study to compare the number of choices in the MC test format used in Indonesian National Examinations.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- Abad, F. J., Olea, J., & Ponsoda, V. (2001). Analysis of the optimum number of alternatives from the Item Response Theory. Psicothema, 13(1), 152-158.
- Allison, D., & Palmer, A. (2010). Language assessment in practice. Oxford University Press.
- Bachman, L.F. (2004). Statistical analyses for language assessment. Cambridge University Press. https://doi.org/10.1017/CBO9780511667350 DOI: https://doi.org/10.1017/CBO9780511667350
- Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple-choice items. Psychological Test and Assessment Modelling, 53(2), 192-211. https://doi.org/10.4304/jltr.2.5.1052-1060 DOI: https://doi.org/10.4304/jltr.2.5.1052-1060
- Bailey, K.M., & Nunan, D. (2009). Exploring second language classroom research: A comprehensive guide. Heinle.
- Boland, R.J., Lester, N.A., & Williams, E. (2010). Writing multiple-choice questions. Acad Psychiatry, 34(4), 310-316. https://doi.org/10.1176/appi.ap.34.4.310 DOI: https://doi.org/10.1176/appi.ap.34.4.310
- Bruno, J. E., & Dirkzwager, A. (1995). Determining the optimal number of alternatives to a multiple-choice test item: An information theoretic perspective. Educational and Psycho-logical Measurement, 55,959-966. https://doi.org/10.1177/0013164495055006004 DOI: https://doi.org/10.1177/0013164495055006004
- Costin, F. (1972). Three-choice versus four-choice items: Implications for Reliability and Validity of objective achievement test. Educational and Psychological Measurement, 32, 1035-1038. https://doi.org/10.1177/001316447203200419 DOI: https://doi.org/10.1177/001316447203200419
- Dehnad, A., Nasser, H., & Hosseini, A. F. (2014). A comparison between three-and four-option multiple choice questions. Procedia-Social and Behavioural Sciences, 98, 398–403. https://doi.org/10.1016/j.sbspro.2014.03.432 DOI: https://doi.org/10.1016/j.sbspro.2014.03.432
- Delgado, A. R., & Prieto, G. (1998). Further evidence favoring three-option items in multiple-choice tests. European Journal of Psychological Assessment, 14, 197-201. https://doi.org/10.1027/1015-5759.14.3.197 DOI: https://doi.org/10.1027/1015-5759.14.3.197
- Dirgeyasa, I. W. (2017). Model answers for yes/no questions from EFL students in public senior high schools. Studies in English Language and Education, 4(2), 235-246. https://doi.org/10.24815/siele.v4i2.7717 DOI: https://doi.org/10.24815/siele.v4i2.7717
- Epstein, R.M. (2007). Medical education: Assessment in medical education. The New England Journal of Medicine, 356(4), 387-396. https://doi.org/10.1056/NEJMra054784 DOI: https://doi.org/10.1056/NEJMra054784
- Frankfort-Nachmias, C. &Nachmias, D. (2007). Study guide for research methods in the social sciences, 7th ed. Macmillan Higher Education.
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334. https://doi.org/10.1207/S15324818AME1503_5
- Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge. https://doi.org/10.4324/9780203850381 DOI: https://doi.org/10.4324/9780203850381
- Hohensinn, C. H., &Kubinger, K. D. (2009). On varying item difficulty by changing the response format for a mathematical competence test. Austrian Journal of Statistics, 38(4), 231-239. https://doi.org/10.17713/ajs.v38i4.276 DOI: https://doi.org/10.17713/ajs.v38i4.276
- Hughes, A. (2003). Testing for language teachers (2nded.). Cambridge University Press.
- Kubinger, K. D., Holocher-Ertl, S., Reif, M., Hohensinn, C., &Frebort, M. (2010). On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to one solution and five distractors item format. International Journal of Selection and Assessment, 18(1), 111-115. https://doi.org/10.1111/j.1468-2389.2010.00493.x DOI: https://doi.org/10.1111/j.1468-2389.2010.00493.x
- Kubiszyn, T., & Borich, G. (2004). Educational testing and measurement: Classroom application and practice (7thed.).John Wiley & Sons, Inc.
- Landrum, R. E., Cashin, J. R., & Thesis, K. S. (1993). More evidence in favor of three-option multiple-choice tests. Educational and Psychological Measurement, 53, 771-778. https://doi.org/10.1177/0013164493053003021 DOI: https://doi.org/10.1177/0013164493053003021
- Lee, H., & Winke, P. (2013). The differences among three-, four-, and five-option-item formats in the context of a high-stakes English-language listening test. Language Testing, 30(1), 99-123. https://doi.org/10.1177/0265532212451235 DOI: https://doi.org/10.1177/0265532212451235
- Masters, J. C., Hulsmeyer, B. S., Pike, M. E., Leichty, K., Miller, M. T., &Verst, A. L. (2001). Assessment of multiple-choice questions in selected test banks accompanying textbooks used in nursing education. Journal of Nursing Education, 40(1), 25-32. DOI: https://doi.org/10.3928/0148-4834-20010101-07
- McCoubrie, P. (2004). Improving the fairness of multiple-choice questions: A literature review. Med Teach,26(8), 709-712. https://doi.org/10.1080/01421590400013495 DOI: https://doi.org/10.1080/01421590400013495
- Owen, S.V., & Froman R.D. (1987). What’s wrong with three-option Multiple-Choice items. Educational and Psychological Measurement, 47, 513-522. https://doi.org/10.1177/0013164487472027 DOI: https://doi.org/10.1177/0013164487472027
- Paharia, P. (2019). A scale development for student satisfaction: A perceived dimension of quality education. Humanities & Social Sciences Reviews, 7(1), 360-366. https://doi.org/10.18510/hssr.2019.7142 DOI: https://doi.org/10.18510/hssr.2019.7142
- Rao, M. M., & Haque, M, I. (2019). A study on the impact of testing on English as a foreign language teaching in a Saudi Arabian university. Humanities & Social Sciences Reviews, 7(2), 58-71. https://doi.org/10.18510/hssr.2019.727 DOI: https://doi.org/10.18510/hssr.2019.727
- Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests (Expanded ed.). University of Chicago Press.
- Redmond, S. P. 1., Hartigan-Rogers, J. A., & Cobbett, S. (2012). High time for a change: Psychometric analysis of multiple-choice questions in nursing. International Journal of Nursing Education Scholarship, 9(1), 1–16. https://doi.org/10.1515/1548-923X.2487 DOI: https://doi.org/10.1515/1548-923X.2487
- Rodriguez, M. (2002). Review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education,15(3), 309-334. https://doi.org/10.1207/S15324818AME1503_5 DOI: https://doi.org/10.1207/S15324818AME1503_5
- Rogers, W.T., & Harley, D. (1999). An empirical comparison of three-and four-choice items and tests: susceptibility to test wiseness and internal consistency reliability. Educational and Psychological Measurement, 59(2), 234-247. https://doi.org/10.1177/00131649921969820 DOI: https://doi.org/10.1177/00131649921969820
- Schneid, S.D, Armour C., Park Y.S., Yudkowsky, R., &Bordage, G. (2014). Reducing the number of options on multiple-choice questions: Response time, psychometrics, and standard-setting. Med Educ., 48, 1020–1027. https://doi.org/10.1111/medu.12525 DOI: https://doi.org/10.1111/medu.12525
- Shizuka, T., Takeuchi, O., Yashima, T., & Yoshizawa, K. (2006). A comparison of three- and four-option English tests for university entrance selection purposes in Japan. Language Testing, 23(1), 35-57. https://doi.org/10.1191/0265532206lt319oa DOI: https://doi.org/10.1191/0265532206lt319oa
- Sidick, J. T., Barrett, G. V., &Doverspike, D. (1994). Three-alternative multiple-choice tests: An attractive option. Personnel Psychology, 47, 829-835. https://doi.org/10.1111/j.1744-6570.1994.tb01579.x DOI: https://doi.org/10.1111/j.1744-6570.1994.tb01579.x
- Sujinah, S., Setyorini, D., Yunianti, S., N., &SaviraIsnah, E. (2019). Developing reading comprehension assessment-based Adobe Flash CC2015 for BIPA Intermediate Level. Humanities & Social Sciences Reviews, 7(3), 460-466. https://doi.org/10.18510/hssr.2019.7367 DOI: https://doi.org/10.18510/hssr.2019.7367
- Susanto, A. (2017). Assessing the relationship between the Vocabulary Level Test (VLT) and Reading Comprehension. Studies in English Language and Education, 4(2), 157-171. https://doi.org/10.24815/siele.v4i2.5118 DOI: https://doi.org/10.24815/siele.v4i2.5118
- Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education, 9(1), 40-47. https://doi.org/10.1186/1472-6920-9-40 DOI: https://doi.org/10.1186/1472-6920-9-40
- Tarrant, M., & Ware, J. (2010). A comparison of the psychometric properties of three-and four-option multiple-choice questions in nursing assessments. Nurse Education Today, 30(6), 539–543. https://doi.org/10.1016/j.nedt.2009.11.002 DOI: https://doi.org/10.1016/j.nedt.2009.11.002
- Tversky, A. (1964). On the optimal number of alternatives at a choice point. Journal of Mathematical Psychology, 1, 386-391. https://doi.org/10.1016/0022-2496(64)90010-0 DOI: https://doi.org/10.1016/0022-2496(64)90010-0
- Vyas, R., & Supe, A. (2008). Multiple choice questions: A literature review on the optimal number of options. The National medical journal of India,21(3), 130-133.
- Wolfe, E. W., & Smith, E. V., Jr. (2007). Instrument development tools and activities for measure validation using Rasch models: Part II – Validation activities. Journal of Applied Measurement, 8, 204-234.
References
Abad, F. J., Olea, J., & Ponsoda, V. (2001). Analysis of the optimum number of alternatives from the Item Response Theory. Psicothema, 13(1), 152-158.
Allison, D., & Palmer, A. (2010). Language assessment in practice. Oxford University Press.
Bachman, L.F. (2004). Statistical analyses for language assessment. Cambridge University Press. https://doi.org/10.1017/CBO9780511667350 DOI: https://doi.org/10.1017/CBO9780511667350
Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple-choice items. Psychological Test and Assessment Modelling, 53(2), 192-211. https://doi.org/10.4304/jltr.2.5.1052-1060 DOI: https://doi.org/10.4304/jltr.2.5.1052-1060
Bailey, K.M., & Nunan, D. (2009). Exploring second language classroom research: A comprehensive guide. Heinle.
Boland, R.J., Lester, N.A., & Williams, E. (2010). Writing multiple-choice questions. Acad Psychiatry, 34(4), 310-316. https://doi.org/10.1176/appi.ap.34.4.310 DOI: https://doi.org/10.1176/appi.ap.34.4.310
Bruno, J. E., & Dirkzwager, A. (1995). Determining the optimal number of alternatives to a multiple-choice test item: An information theoretic perspective. Educational and Psycho-logical Measurement, 55,959-966. https://doi.org/10.1177/0013164495055006004 DOI: https://doi.org/10.1177/0013164495055006004
Costin, F. (1972). Three-choice versus four-choice items: Implications for Reliability and Validity of objective achievement test. Educational and Psychological Measurement, 32, 1035-1038. https://doi.org/10.1177/001316447203200419 DOI: https://doi.org/10.1177/001316447203200419
Dehnad, A., Nasser, H., & Hosseini, A. F. (2014). A comparison between three-and four-option multiple choice questions. Procedia-Social and Behavioural Sciences, 98, 398–403. https://doi.org/10.1016/j.sbspro.2014.03.432 DOI: https://doi.org/10.1016/j.sbspro.2014.03.432
Delgado, A. R., & Prieto, G. (1998). Further evidence favoring three-option items in multiple-choice tests. European Journal of Psychological Assessment, 14, 197-201. https://doi.org/10.1027/1015-5759.14.3.197 DOI: https://doi.org/10.1027/1015-5759.14.3.197
Dirgeyasa, I. W. (2017). Model answers for yes/no questions from EFL students in public senior high schools. Studies in English Language and Education, 4(2), 235-246. https://doi.org/10.24815/siele.v4i2.7717 DOI: https://doi.org/10.24815/siele.v4i2.7717
Epstein, R.M. (2007). Medical education: Assessment in medical education. The New England Journal of Medicine, 356(4), 387-396. https://doi.org/10.1056/NEJMra054784 DOI: https://doi.org/10.1056/NEJMra054784
Frankfort-Nachmias, C. &Nachmias, D. (2007). Study guide for research methods in the social sciences, 7th ed. Macmillan Higher Education.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334. https://doi.org/10.1207/S15324818AME1503_5
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge. https://doi.org/10.4324/9780203850381 DOI: https://doi.org/10.4324/9780203850381
Hohensinn, C. H., &Kubinger, K. D. (2009). On varying item difficulty by changing the response format for a mathematical competence test. Austrian Journal of Statistics, 38(4), 231-239. https://doi.org/10.17713/ajs.v38i4.276 DOI: https://doi.org/10.17713/ajs.v38i4.276
Hughes, A. (2003). Testing for language teachers (2nded.). Cambridge University Press.
Kubinger, K. D., Holocher-Ertl, S., Reif, M., Hohensinn, C., &Frebort, M. (2010). On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to one solution and five distractors item format. International Journal of Selection and Assessment, 18(1), 111-115. https://doi.org/10.1111/j.1468-2389.2010.00493.x DOI: https://doi.org/10.1111/j.1468-2389.2010.00493.x
Kubiszyn, T., & Borich, G. (2004). Educational testing and measurement: Classroom application and practice (7thed.).John Wiley & Sons, Inc.
Landrum, R. E., Cashin, J. R., & Thesis, K. S. (1993). More evidence in favor of three-option multiple-choice tests. Educational and Psychological Measurement, 53, 771-778. https://doi.org/10.1177/0013164493053003021 DOI: https://doi.org/10.1177/0013164493053003021
Lee, H., & Winke, P. (2013). The differences among three-, four-, and five-option-item formats in the context of a high-stakes English-language listening test. Language Testing, 30(1), 99-123. https://doi.org/10.1177/0265532212451235 DOI: https://doi.org/10.1177/0265532212451235
Masters, J. C., Hulsmeyer, B. S., Pike, M. E., Leichty, K., Miller, M. T., &Verst, A. L. (2001). Assessment of multiple-choice questions in selected test banks accompanying textbooks used in nursing education. Journal of Nursing Education, 40(1), 25-32. DOI: https://doi.org/10.3928/0148-4834-20010101-07
McCoubrie, P. (2004). Improving the fairness of multiple-choice questions: A literature review. Med Teach,26(8), 709-712. https://doi.org/10.1080/01421590400013495 DOI: https://doi.org/10.1080/01421590400013495
Owen, S.V., & Froman R.D. (1987). What’s wrong with three-option Multiple-Choice items. Educational and Psychological Measurement, 47, 513-522. https://doi.org/10.1177/0013164487472027 DOI: https://doi.org/10.1177/0013164487472027
Paharia, P. (2019). A scale development for student satisfaction: A perceived dimension of quality education. Humanities & Social Sciences Reviews, 7(1), 360-366. https://doi.org/10.18510/hssr.2019.7142 DOI: https://doi.org/10.18510/hssr.2019.7142
Rao, M. M., & Haque, M, I. (2019). A study on the impact of testing on English as a foreign language teaching in a Saudi Arabian university. Humanities & Social Sciences Reviews, 7(2), 58-71. https://doi.org/10.18510/hssr.2019.727 DOI: https://doi.org/10.18510/hssr.2019.727
Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests (Expanded ed.). University of Chicago Press.
Redmond, S. P. 1., Hartigan-Rogers, J. A., & Cobbett, S. (2012). High time for a change: Psychometric analysis of multiple-choice questions in nursing. International Journal of Nursing Education Scholarship, 9(1), 1–16. https://doi.org/10.1515/1548-923X.2487 DOI: https://doi.org/10.1515/1548-923X.2487
Rodriguez, M. (2002). Review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education,15(3), 309-334. https://doi.org/10.1207/S15324818AME1503_5 DOI: https://doi.org/10.1207/S15324818AME1503_5
Rogers, W.T., & Harley, D. (1999). An empirical comparison of three-and four-choice items and tests: susceptibility to test wiseness and internal consistency reliability. Educational and Psychological Measurement, 59(2), 234-247. https://doi.org/10.1177/00131649921969820 DOI: https://doi.org/10.1177/00131649921969820
Schneid, S.D, Armour C., Park Y.S., Yudkowsky, R., &Bordage, G. (2014). Reducing the number of options on multiple-choice questions: Response time, psychometrics, and standard-setting. Med Educ., 48, 1020–1027. https://doi.org/10.1111/medu.12525 DOI: https://doi.org/10.1111/medu.12525
Shizuka, T., Takeuchi, O., Yashima, T., & Yoshizawa, K. (2006). A comparison of three- and four-option English tests for university entrance selection purposes in Japan. Language Testing, 23(1), 35-57. https://doi.org/10.1191/0265532206lt319oa DOI: https://doi.org/10.1191/0265532206lt319oa
Sidick, J. T., Barrett, G. V., &Doverspike, D. (1994). Three-alternative multiple-choice tests: An attractive option. Personnel Psychology, 47, 829-835. https://doi.org/10.1111/j.1744-6570.1994.tb01579.x DOI: https://doi.org/10.1111/j.1744-6570.1994.tb01579.x
Sujinah, S., Setyorini, D., Yunianti, S., N., &SaviraIsnah, E. (2019). Developing reading comprehension assessment-based Adobe Flash CC2015 for BIPA Intermediate Level. Humanities & Social Sciences Reviews, 7(3), 460-466. https://doi.org/10.18510/hssr.2019.7367 DOI: https://doi.org/10.18510/hssr.2019.7367
Susanto, A. (2017). Assessing the relationship between the Vocabulary Level Test (VLT) and Reading Comprehension. Studies in English Language and Education, 4(2), 157-171. https://doi.org/10.24815/siele.v4i2.5118 DOI: https://doi.org/10.24815/siele.v4i2.5118
Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education, 9(1), 40-47. https://doi.org/10.1186/1472-6920-9-40 DOI: https://doi.org/10.1186/1472-6920-9-40
Tarrant, M., & Ware, J. (2010). A comparison of the psychometric properties of three-and four-option multiple-choice questions in nursing assessments. Nurse Education Today, 30(6), 539–543. https://doi.org/10.1016/j.nedt.2009.11.002 DOI: https://doi.org/10.1016/j.nedt.2009.11.002
Tversky, A. (1964). On the optimal number of alternatives at a choice point. Journal of Mathematical Psychology, 1, 386-391. https://doi.org/10.1016/0022-2496(64)90010-0 DOI: https://doi.org/10.1016/0022-2496(64)90010-0
Vyas, R., & Supe, A. (2008). Multiple choice questions: A literature review on the optimal number of options. The National medical journal of India,21(3), 130-133.
Wolfe, E. W., & Smith, E. V., Jr. (2007). Instrument development tools and activities for measure validation using Rasch models: Part II – Validation activities. Journal of Applied Measurement, 8, 204-234.