Coreference Annotation

Annotation scheme

Event triggers and participants in our corpus are annotated within the boundaries of a single sentence. However, resolving anaphoric expressions that act as event participants to their coreferent NEs in other sentences is an important step to ensure the accurate extraction and/or interpretation of events.

Our coreference annotation links anaphoric event participants to their antecedent NEs, as follows:

  • If the only reference to an event participant within an event-containing sentence is an anaphoric expression that is too general to be annotated as an NE (e.g. it, this drug, such disorders, etc.), then the expression is annotated using the special category Coreferring_mention, as long as BOTH of the following hold:
    • There is an annotated NE in a nearby sentence to which the anaphoric expression refers.
    • Linking the Coreferring_mention as a participant of the event results in an event that is valid (see page on Event annotation for more details)
  • The Coreferring_mention annotation and the previously annotated NE are linked using a Coreference relation.

A total of 212 coreference annotations have been annotated in the corpus.

Figure 1 shows an example of coreference annotation.

Co-reference annotation example


Since coreference is annotated at the level of event participants, we calculate agreement for coreference annotation as follows:

  • We examine events that are agreed by both annotators, according to the relaxed criterion for event matching (see Event Annotation page for more details)
  • For the agreed events, we consider participants that have been annotated as Coreferring_mention by at least one of the annotators. A match is defined as a case in which the following conditions hold:
    • Both annotators have identified Coreferring_mention spans that overlap with each other AND
    • The Coreferring_mention spans are assigned the same semantic role in the matched events AND
    • The Coreferring_mention spans annotated by each annotator are linked to the same NE in a neighbouring sentence via a Coreference relation (i.e. the linked NEs must have overlapping spans, and must be assigned the same NE category)
By applying these criteria, we achieved an agreement rate of 50.91 F-Score.