This paper reports on the methodological procedures implemented to address issues of reliability in manual annotations as part of an ongoing PhD project on the Acquisition of Tense and Aspect in learners of English as L3. A combination of multiple procedures to assess inter- and intra-rater reliability coefficients among several coders proved to be a useful method to reflect on possible cases of low agreement in problematic layers of annotations and to highlight emerging annotation patterns among the coders. Moreover, the tests designed in our study aided in identifying potential problems associated with developing instruments and guidelines that explained the coding task, pinpointing particularly challenging categories, and contributing to improving overall accuracy and transparency in the annotation process.
Assessing Inter- and Intra-Rater Reliability in Multi-Layer Annotations: A study on Tense and Aspect in learners of English as L3
Lopopolo, Olga
;Zanda, Fabio
2024-01-01
Abstract
This paper reports on the methodological procedures implemented to address issues of reliability in manual annotations as part of an ongoing PhD project on the Acquisition of Tense and Aspect in learners of English as L3. A combination of multiple procedures to assess inter- and intra-rater reliability coefficients among several coders proved to be a useful method to reflect on possible cases of low agreement in problematic layers of annotations and to highlight emerging annotation patterns among the coders. Moreover, the tests designed in our study aided in identifying potential problems associated with developing instruments and guidelines that explained the coding task, pinpointing particularly challenging categories, and contributing to improving overall accuracy and transparency in the annotation process.File | Dimensione | Formato | |
---|---|---|---|
7_1.2. Lopopolo et Zanda.pdf
non disponibili
Tipologia:
Versione Editoriale (PDF)
Licenza:
NON PUBBLICO - Accesso chiuso
Dimensione
1.12 MB
Formato
Adobe PDF
|
1.12 MB | Adobe PDF | Visualizza/Apri Richiedi una copia |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.