This paper reports on the methodological procedures implemented to address issues of reliability in manual annotations as part of an ongoing PhD project on the Acquisition of Tense and Aspect in learners of English as L3. A combination of multiple procedures to assess inter- and intra-rater reliability coefficients among several coders proved to be a useful method to reflect on possible cases of low agreement in problematic layers of annotations and to highlight emerging annotation patterns among the coders. Moreover, the tests designed in our study aided in identifying potential problems associated with developing instruments and guidelines that explained the coding task, pinpointing particularly challenging categories, and contributing to improving overall accuracy and transparency in the annotation process.

Assessing Inter- and Intra-Rater Reliability in Multi-Layer Annotations: A study on Tense and Aspect in learners of English as L3

Lopopolo, Olga
;
Zanda, Fabio
2024-01-01

Abstract

This paper reports on the methodological procedures implemented to address issues of reliability in manual annotations as part of an ongoing PhD project on the Acquisition of Tense and Aspect in learners of English as L3. A combination of multiple procedures to assess inter- and intra-rater reliability coefficients among several coders proved to be a useful method to reflect on possible cases of low agreement in problematic layers of annotations and to highlight emerging annotation patterns among the coders. Moreover, the tests designed in our study aided in identifying potential problems associated with developing instruments and guidelines that explained the coding task, pinpointing particularly challenging categories, and contributing to improving overall accuracy and transparency in the annotation process.
2024
978-2-39061-525-5
inter rater reliability, intra rater reliability, Tense and Aspect, English as L3
File in questo prodotto:
File Dimensione Formato  
7_1.2. Lopopolo et Zanda.pdf

non disponibili

Tipologia: Versione Editoriale (PDF)
Licenza: NON PUBBLICO - Accesso chiuso
Dimensione 1.12 MB
Formato Adobe PDF
1.12 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.12071/46388
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
social impact