Authors retain the copyright without restrictions for their published content in this journal. HSSR is a SHERPA ROMEO Green Journal.
Publishing License
This is an open-access article distributed under the terms of
APPLYING ITEM RESPONSES THEORY FOR MEASURING STUDENT’S ABILITY IN ACADEMIC SPEAKING
Corresponding Author(s) : Memet Sudaryanto
Humanities & Social Sciences Reviews,
Vol. 8 No. 2 (2020): March
Abstract
Purpose of the study: Every human being is required to be skilled at communicating, skilled at expressing thoughts, ideas, and feelings. This research aims to explain student’s ability in academic speaking based on the framework of item responses theory.
Methodology: This research uses a mix method research approach; qualitative and quantitative approaches are used together to answer the formulated problems. A qualitative approach is used for digging up information about the needs to develop the speaking test. The try out subjects was 25 university student taking Bahasa Indonesia subject; while 125 students were used as the measurement subject. The data were students’ responses to the speaking test which was scored by two people (rater). The reliability of the instrument was estimated by the Generalizability theory. Rasch model analysis was used to estimate the item parameter; while the Maximum Likelihood was used to estimate the students’ ability.
Main Findings: The value of 𜎠̂(ð‘œ) s influenced by the similarity to the average mean score observed in academic speaking. The value of 𜎠̂(ð‘) and 𜎠̂(ð‘œ) suggests that the distribution of variability in person and item is the same and high. A sufficiently large value of 𜎠̂(ð‘i) implies the fact that the value involves all residual sources for a variance. Balance alternatives, weigh consequently and decide rationally.
Applications of this study: Based on the design proposed for the Indonesian Language Proficiency assessment, other generalizability design (G-Design) used is a cross design because each student (p) becomes the object of observation of two observers (r) who both assess four aspects of observation/indicator. It is used to determine the reliability (i.e., reproducibility) of measurements under specific conditions in academic speaking.
Novelty/Originality of this study: Some rater qualifications that must be met include the process of gathering and using the appropriateness of background information before assessing. Measuring the ability of students in academic speaking by applying G theory and conducting an IRT analysis approach needs.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- Andrade, M. S. (2006). International students in English-speaking universities: Adjustment factors. Journal of Research in International Education. https://doi.org/10.1177/1475240906065589 DOI: https://doi.org/10.1177/1475240906065589
- Brennan, R. L. (2010). Generalizability theory. In International Encyclopedia of Education. https://doi.org/10.1016/B978-0-08-044894-7.00246-3 DOI: https://doi.org/10.1016/B978-0-08-044894-7.00246-3
- Brooks-Lewis, K. A. (2009). Adult learners’ perceptions of the incorporation of their L1 in foreign language teaching and learning. Applied Linguistics. https://doi.org/10.1093/applin/amn051 DOI: https://doi.org/10.1093/applin/amn051
- Cunningsworth, A., & Horner, D. (1985). The role of simulations in the development of communication strategies. System. https://doi.org/10.1016/0346-251X(85)90035-1 DOI: https://doi.org/10.1016/0346-251X(85)90035-1
- de Beaugrande, R. (2006). Critical discourse analysis: History, ideology, methodology. Studies in Language and Capitalism. https://doi.org/diskursanalyse; qualitative methode; ideologie
- Fraley, R. C., Waller, N. G., & Brennan, K. A. (2000). An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology. https://doi.org/10.1037/0022-3514.78.2.350 DOI: https://doi.org/10.1037/0022-3514.78.2.350
- Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability Estimation in a Multilevel Confirmatory Factor Analysis Framework. 19(1), 72–91. https://doi.org/10.1037/a0032138 DOI: https://doi.org/10.1037/a0032138
- Hambleton, R. K., & Swaminathan, H. (1985). Item response theory : principles and applications. In Evaluation in education and human services; https://doi.org/10.1017/CBO9781107415324.004 DOI: https://doi.org/10.1017/CBO9781107415324.004
- Horwitz, E. (2001). Language anxiety and achievement. Annual Review of Applied Linguistics. https://doi.org/10.1017/S0267190501000071 DOI: https://doi.org/10.1017/S0267190501000071
- Kamata, A., Turhan, A., & Darandari, E. (2003). Estimating reliability for multidimensional composite scale scores. Annual Meeting of American Educational Research Association, Chicago, April 2003.
- Kuzar, R. (2003). Constructions: A construction grammar approach to argument structure. Journal of Pragmatics. https://doi.org/10.1016/S0378-2166(97)81937-6 DOI: https://doi.org/10.1016/S0378-2166(97)81937-6
- Laufer, B., & Aviad-Levitzky, T. (2017). What Type of Vocabulary Knowledge Predicts Reading Comprehension: Word Meaning Recall or Word Meaning Recognition? Modern Language Journal. https://doi.org/10.1111/modl.12431
- Markhamah, Ngalim, A., Muinudinillah Basri, M., & Sabardila, A. (2017). Comparison of Personal Pronoun between Arabic and Its Indonesian Translation of Koran. International Journal of Applied Linguistics and English Literature. https://doi.org/10.7575/aiac.ijalel.v.6n.5p.238 DOI: https://doi.org/10.7575/aiac.ijalel.v.6n.5p.238
- Matt, G. E., & Sklar, M. (2015). Generalizability Theory. In International Encyclopedia of the Social & Behavioral Sciences: Second Edition. https://doi.org/10.1016/B978-0-08-097086-8.44027-4 DOI: https://doi.org/10.1016/B978-0-08-097086-8.44027-4
- Mattarima, K., & Rahim Hamdan, A. (2011). The Teaching Constraints of English as a Foreign Language in Indonesia: The Context of School Based Curriculum. Sosiohumanika.
- Miller, N. (2013). Measuring up to speech intelligibility. International Journal of Language and Communication Disorders. https://doi.org/10.1111/1460-6984.12061 DOI: https://doi.org/10.1111/1460-6984.12061
- Moore, K. D. ((2005). Effective instructional strategies: From theory to practice. In Sage Publications.
- Nash, G., Crimmins, G., & Oprescu, F. (2016). If first-year students are afraid of public speaking assessments what can teachers do to alleviate such anxiety? Assessment and Evaluation in Higher Education. https://doi.org/10.1080/02602938.2015.1032212 DOI: https://doi.org/10.1080/02602938.2015.1032212
- Nurgiyantoro, B. (2009). Penilaian dalam pengajaran bahasa dan sastra. Yogyakarta: BPFE.
- Oradee, T. (2013). Developing Speaking Skills Using Three Communicative Activities (Discussion, Problem-Solving, andRole-Playing). International Journal of Social Science and Humanity. https://doi.org/10.7763/IJSSH.2012.V2.164 DOI: https://doi.org/10.7763/IJSSH.2012.V2.164
- Paul, R., & Smith, R. L. (1993). Narrative skills in 4-year-olds with normal, impaired, and late- developing language. Journal of Speech and Hearing Research. https://doi.org/10.1044/jshr.3603.592 DOI: https://doi.org/10.1044/jshr.3603.592
- Penev, S., & Raykov, T. (2006). On the relationship between maximal reliability and maximal validity of linear composites. Multivariate Behavioral Research. https://doi.org/10.1207/s15327906mbr4102_1 DOI: https://doi.org/10.1207/s15327906mbr4102_1
- Pyrik, J. (2015). Communicating Risk. In Intelligence Communication in the Digital Era: Transforming Security, Defence and Business. https://doi.org/10.1057/9781137523792_4 DOI: https://doi.org/10.1057/9781137523792_4
- Rahmawati, L. E., Suwandi, S., Saddhono, K., & Setiawan, B. (2019). Need analysis on the development of writing competency test for foreign university students. Humanities and Social Sciences Reviews, 7(3). https://doi.org/10.18510/hssr.2019.7368 DOI: https://doi.org/10.18510/hssr.2019.7368
- Retnawati, H. (2008). Validitas dan reliabilitas konstruk skor tes kemampuan calon mahasiswa. (1), 126–135.
- Retnawati, H. (2016). Proving content validity of self-regulated learning scale (The comparison of Aiken index and expanded Gregory index). Research and Evaluation in Education. https://doi.org/10.21831/reid.v2i2.11029 DOI: https://doi.org/10.21831/reid.v2i2.11029
- Retnawati, H. (2017). Learning trajectory of item response theory course using multiple softwares. Olympiads in Informatics. https://doi.org/10.15388/ioi.2017.10 DOI: https://doi.org/10.15388/ioi.2017.10
- Saddhono, K. (2015). Integrating culture in Indonesian language learning for foreign speakers at Indonesian universities. Journal of Language and Literature. https://doi.org/10.7813/jll.2015/6-2/58
- Sudaryanto, M., Mardapi, D., & Hadi, S. (2019). How foreign speakers implement their strategies to listen indonesian language? Journal of Advanced Research in Dynamical and Control Systems.
- Sudaryanto, M., Saddhono, K., Wahyono, H., Widiatmi, T., Ino, L., Hidayat, H., … Pramono, P. (2019). Indonesian as a Foreign Language: Standard Setting and Materials Development Issues. 1st Workshop on Environmental Science, Society, and Technology, WESTECH, 178–184. https://doi.org/10.4108/eai.8-12-2018.2284065 DOI: https://doi.org/10.4108/eai.8-12-2018.2284065
- Townsend, S. S. M., Kim, H. S., & Mesquita, B. (2014). Are You Feeling What I’m Feeling? Emotional Similarity Buffers Stress. Social Psychological and Personality Science. https://doi.org/10.1177/1948550613511499 DOI: https://doi.org/10.1177/1948550613511499
- Winarni, R., Slamet, S. Y., & Saddhono, K. (2018). Development of Indonesian literature textbook with character education through information and communication technology (ICT) learning based. International Journal of Engineering and Technology(UAE). https://doi.org/10.5465/amr.2009.35713291 DOI: https://doi.org/10.5465/amr.2009.35713291
- Zyoud, M. M. (2016). Theoretical Perspective on How To Develop Speaking Skill. An International Multidisciplinary Journal.
References
Andrade, M. S. (2006). International students in English-speaking universities: Adjustment factors. Journal of Research in International Education. https://doi.org/10.1177/1475240906065589 DOI: https://doi.org/10.1177/1475240906065589
Brennan, R. L. (2010). Generalizability theory. In International Encyclopedia of Education. https://doi.org/10.1016/B978-0-08-044894-7.00246-3 DOI: https://doi.org/10.1016/B978-0-08-044894-7.00246-3
Brooks-Lewis, K. A. (2009). Adult learners’ perceptions of the incorporation of their L1 in foreign language teaching and learning. Applied Linguistics. https://doi.org/10.1093/applin/amn051 DOI: https://doi.org/10.1093/applin/amn051
Cunningsworth, A., & Horner, D. (1985). The role of simulations in the development of communication strategies. System. https://doi.org/10.1016/0346-251X(85)90035-1 DOI: https://doi.org/10.1016/0346-251X(85)90035-1
de Beaugrande, R. (2006). Critical discourse analysis: History, ideology, methodology. Studies in Language and Capitalism. https://doi.org/diskursanalyse; qualitative methode; ideologie
Fraley, R. C., Waller, N. G., & Brennan, K. A. (2000). An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology. https://doi.org/10.1037/0022-3514.78.2.350 DOI: https://doi.org/10.1037/0022-3514.78.2.350
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability Estimation in a Multilevel Confirmatory Factor Analysis Framework. 19(1), 72–91. https://doi.org/10.1037/a0032138 DOI: https://doi.org/10.1037/a0032138
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory : principles and applications. In Evaluation in education and human services; https://doi.org/10.1017/CBO9781107415324.004 DOI: https://doi.org/10.1017/CBO9781107415324.004
Horwitz, E. (2001). Language anxiety and achievement. Annual Review of Applied Linguistics. https://doi.org/10.1017/S0267190501000071 DOI: https://doi.org/10.1017/S0267190501000071
Kamata, A., Turhan, A., & Darandari, E. (2003). Estimating reliability for multidimensional composite scale scores. Annual Meeting of American Educational Research Association, Chicago, April 2003.
Kuzar, R. (2003). Constructions: A construction grammar approach to argument structure. Journal of Pragmatics. https://doi.org/10.1016/S0378-2166(97)81937-6 DOI: https://doi.org/10.1016/S0378-2166(97)81937-6
Laufer, B., & Aviad-Levitzky, T. (2017). What Type of Vocabulary Knowledge Predicts Reading Comprehension: Word Meaning Recall or Word Meaning Recognition? Modern Language Journal. https://doi.org/10.1111/modl.12431
Markhamah, Ngalim, A., Muinudinillah Basri, M., & Sabardila, A. (2017). Comparison of Personal Pronoun between Arabic and Its Indonesian Translation of Koran. International Journal of Applied Linguistics and English Literature. https://doi.org/10.7575/aiac.ijalel.v.6n.5p.238 DOI: https://doi.org/10.7575/aiac.ijalel.v.6n.5p.238
Matt, G. E., & Sklar, M. (2015). Generalizability Theory. In International Encyclopedia of the Social & Behavioral Sciences: Second Edition. https://doi.org/10.1016/B978-0-08-097086-8.44027-4 DOI: https://doi.org/10.1016/B978-0-08-097086-8.44027-4
Mattarima, K., & Rahim Hamdan, A. (2011). The Teaching Constraints of English as a Foreign Language in Indonesia: The Context of School Based Curriculum. Sosiohumanika.
Miller, N. (2013). Measuring up to speech intelligibility. International Journal of Language and Communication Disorders. https://doi.org/10.1111/1460-6984.12061 DOI: https://doi.org/10.1111/1460-6984.12061
Moore, K. D. ((2005). Effective instructional strategies: From theory to practice. In Sage Publications.
Nash, G., Crimmins, G., & Oprescu, F. (2016). If first-year students are afraid of public speaking assessments what can teachers do to alleviate such anxiety? Assessment and Evaluation in Higher Education. https://doi.org/10.1080/02602938.2015.1032212 DOI: https://doi.org/10.1080/02602938.2015.1032212
Nurgiyantoro, B. (2009). Penilaian dalam pengajaran bahasa dan sastra. Yogyakarta: BPFE.
Oradee, T. (2013). Developing Speaking Skills Using Three Communicative Activities (Discussion, Problem-Solving, andRole-Playing). International Journal of Social Science and Humanity. https://doi.org/10.7763/IJSSH.2012.V2.164 DOI: https://doi.org/10.7763/IJSSH.2012.V2.164
Paul, R., & Smith, R. L. (1993). Narrative skills in 4-year-olds with normal, impaired, and late- developing language. Journal of Speech and Hearing Research. https://doi.org/10.1044/jshr.3603.592 DOI: https://doi.org/10.1044/jshr.3603.592
Penev, S., & Raykov, T. (2006). On the relationship between maximal reliability and maximal validity of linear composites. Multivariate Behavioral Research. https://doi.org/10.1207/s15327906mbr4102_1 DOI: https://doi.org/10.1207/s15327906mbr4102_1
Pyrik, J. (2015). Communicating Risk. In Intelligence Communication in the Digital Era: Transforming Security, Defence and Business. https://doi.org/10.1057/9781137523792_4 DOI: https://doi.org/10.1057/9781137523792_4
Rahmawati, L. E., Suwandi, S., Saddhono, K., & Setiawan, B. (2019). Need analysis on the development of writing competency test for foreign university students. Humanities and Social Sciences Reviews, 7(3). https://doi.org/10.18510/hssr.2019.7368 DOI: https://doi.org/10.18510/hssr.2019.7368
Retnawati, H. (2008). Validitas dan reliabilitas konstruk skor tes kemampuan calon mahasiswa. (1), 126–135.
Retnawati, H. (2016). Proving content validity of self-regulated learning scale (The comparison of Aiken index and expanded Gregory index). Research and Evaluation in Education. https://doi.org/10.21831/reid.v2i2.11029 DOI: https://doi.org/10.21831/reid.v2i2.11029
Retnawati, H. (2017). Learning trajectory of item response theory course using multiple softwares. Olympiads in Informatics. https://doi.org/10.15388/ioi.2017.10 DOI: https://doi.org/10.15388/ioi.2017.10
Saddhono, K. (2015). Integrating culture in Indonesian language learning for foreign speakers at Indonesian universities. Journal of Language and Literature. https://doi.org/10.7813/jll.2015/6-2/58
Sudaryanto, M., Mardapi, D., & Hadi, S. (2019). How foreign speakers implement their strategies to listen indonesian language? Journal of Advanced Research in Dynamical and Control Systems.
Sudaryanto, M., Saddhono, K., Wahyono, H., Widiatmi, T., Ino, L., Hidayat, H., … Pramono, P. (2019). Indonesian as a Foreign Language: Standard Setting and Materials Development Issues. 1st Workshop on Environmental Science, Society, and Technology, WESTECH, 178–184. https://doi.org/10.4108/eai.8-12-2018.2284065 DOI: https://doi.org/10.4108/eai.8-12-2018.2284065
Townsend, S. S. M., Kim, H. S., & Mesquita, B. (2014). Are You Feeling What I’m Feeling? Emotional Similarity Buffers Stress. Social Psychological and Personality Science. https://doi.org/10.1177/1948550613511499 DOI: https://doi.org/10.1177/1948550613511499
Winarni, R., Slamet, S. Y., & Saddhono, K. (2018). Development of Indonesian literature textbook with character education through information and communication technology (ICT) learning based. International Journal of Engineering and Technology(UAE). https://doi.org/10.5465/amr.2009.35713291 DOI: https://doi.org/10.5465/amr.2009.35713291
Zyoud, M. M. (2016). Theoretical Perspective on How To Develop Speaking Skill. An International Multidisciplinary Journal.