Relation Annotation

Annotation scheme

We define two types of binary relations that encode static properties between NEs, as a means further enrich the dynamic information that is encoded through event structures. These two relation types are defined in Table 1.

Table 1. Types of Relations annotated
Relation typeDescription
Subject_Disorder Connects Subject phrases and Disorders, when the mentioned disorders correspond to complaints suffered by the subject(s) at the time when pharmacological substances are administered
is_equivalent Allows links to be established between NEs that constitute alternative names for the same concept within the same sentence. Equivalences may correspond to full drug names/disorders and their abbreviations, to generic drug names and their corresponding brand names or synonyms, etc

Relation annotation statistics

All instances of the relations defined in Table 1 were annotated in all abstracts in the corpus. The total number of annotated relations of each type are shown in Table 2

Table 2. Statistics of NE Mentions
Relation TypeTotal number of annotated relations
Subject_Disorder 636


The relation annotations were undertaken by annotators with domain expertise. The quality and consistency of the annotations were verified through the calculation of inter-annotator agreement (IAA) on one quarter of the complete corpus (i.e., 150 abstracts). We calculated IAA in terms of F-Score, as shown in Table 3.

Table 3. Inter-annotator agreement rates for relations (F-score)
Relation TypeAgreement Rate
Subject_Disorder 69.3
TOTAL 72.6