Measuring same construct in two languages; how much comparable?

Junghee Byun, Jungyeon Koo

Comment [RH1]: question NP or question whether CP Comment [RH2]: What does?

Since Cummins (1981) proposed Interdependence Hypothesis and common cognitive-academic language proficiency, there have been studies that question Contrastive Analysis (CA) to see L1 rhetorical knowledge as interfering L2 writing and assert L1 discourse competence facilitating L2 writing literacy (Cumming, 1988; Hirose, 2003; Ferris & Hedgecock, 2005). It suggests much to Korean EFL context where English learners have difficulty writing their thoughts in L1 and L2 due to lack of formal Korean writing classes and absence of Korean high-stake tests (Choi, 2013). This contrasts with English writing which has emphasized formal instruction as a high-stake test. Motivated by the interconnectedness theory and the need in Korean EFL context, the paper explores construct comparability to assess Korean learners discourse competence in their L1 and L2, English, in aspects of an examinees ability, task, rater and scale. For this study, 31 college students at high English proficiency level (TOEFL iBT 106 in on average) write a pair of argumentative essays first in Korean and later in English later given the same prompt. An analytic scale is developed by extracting common discourse features from the two sources of scoring rubrics (Weigle, 2002; Yeo, 2002). Then, six raters score the 31 pairs of writings by the same authors on a scale of 1~5 on four criteria - task fulfillment (TF), organization (ORG), elaboration of idea (ELB) and coherence (COH). The scoring data are analyzed by a multi-facet Rasch model using FACET, a two-way mixed-repeated measures ANOVA and multiple regression. Results shows that two thirds of examinees are in the same ability group between the upper and lower L1 and L2, which is encouraging to test comparability. High reliability of the two tests is evidenced by Chronbach Alpha coefficients over 0.90 and the high reliability estimates of the abilities of examinees over 0.90 logits. The tests have the same task difficulty of a standard essay to for the examinees (0 logit) and lenient raters without any extreme raters. The four criteria fit to the model consistently increase in logit as the markings rise from 1 to 5 despite less scale equivalence in the criteria except TF. The sum scores across the six raters are correlated in the range of .38~ .65 (p <.05). There is no mean difference within language group while significant mean difference emerges in COH between the two language groups (F= 5.50, partial eta square = .084 p=.022). The Llow mean of COH in the English test might result from the laborious cognitive process for L2 writers to experience in handling logical flow and language switch altogether (Silva, 1992). Multiple regression reports ORG and TF most explain the Korean and English sum scores each. This may imply the Korean raters prioritize formality whereas the English content and the requirements for task completion. It is cautiously concluded that the tests similarly assess discourse competence for English and Korean argumentative writings, emphasizing similar discourse features unlike CA. This study hopefully will contribute to reevaluating the L1 discourse competence to step forward macro-level of L2 writing.

Comment [RH3]: Awkward. Comment [RH4]: What theory? Comment [RH5]: What need?

Comment [RH6]: Is this apposition? If s then perhaps i.e. is needed. Comment [RH7]: with or of Comment [RH8]: Unclear, perhaps unnecessary

Comment [RH9]: Perhaps according to Comment [RH10]: Or the Comment [RH11]: Maybe the Comment [RH12]: Perhaps this should be a separate sentence. Comment [RH13]:

Comment [RH14]: What does this mea

Comment [RH15]: This sentence alread has a finite verb, perhaps you mean increasing.

Comment [RH16]: I cannot understand what this means. Comment [RH17]: Awkward structure. The cognitive process experiences something? Maybe the laborious cognitive process of L2 writers experiencing handling logical flow Comment [RH18]: Awkward. Does this explains English? If so, then you mean word is needed. Comment [RH19]: Ellipsis is awkward Comment [RH20]: Perhaps the Comment [RH21]: I dont understand.

mean that ORG explains Korean and TF

write respectively. If not, then some oth