Relation Annotation in CHF

Annotation scheme

In the narrative EHR reports of PhenoCHF, three types of relations involving entity mentions have been annotated. Compared to entity mention annotation, which identifies mentions of concepts occurring in text, relation annotation encodes some of the more complex pieces of information expressed in text. Specifically, a sentence will often describe how different concepts are are linked toogether in particular ways, or will provide a particular interpretation of an entity.

Table 1 provides details of the three types of relations that have been annotated in PhenoCHF (narrative EHR reports only). Please see the Entity Mention annotation page for descriptions of the different types of entities involved in the relationships. All annotated relationships occur within the scope of a single sentence. Two of these relationships involve pairs of entity mentions which, according to the information provided in the sentence, are associated with each other in specific ways. The semantic label assigned to the relation determines the nature of the association between the two entity mentions. The third relationship (Negate) is annotated when the context of the sentence alters the default interpretation of an entity (i.e., it becomes negated).

Table 1. Types of relations annotated in PhenoCHF
Relation TypeDescriptionFirst entity type(s)Second entity type(s)Example
CausalityThe concept referred to by the first entity mention is responsible for the concept refered to by the second entity mention Chief complaint


Risk Factor

traditional Risk factor
traditional Risk Factor

FindingThe mentioned organ is associated with the manifestation or abnormal variation that is observed during the diagnosis process.OrganSign & Symptom
Negate A word or phrase denoting negation (a polarity cue) is annotated and linked to the mention of the condition that it negatesPolarity cueFinding


traditional risk factor

The two relationships between pairs of entity mentions are are based on relationships in the UMLS semantic network:

  • Causality - based on the causes relation in the UMLS semantic network that holds between two diseases or a disease and a pathologic function
  • Finding - based on the manifestation relation in the UMLS semantic network that holds between a sign or symptom and a body part or organ

Relation annotation statistics

All relationships of the types shown in Table 1 were annotated in all narrative EHR records. The total counts of each type of relationship annotated are shown in Table 2

Table 2. Statitics of Relation Annotations in PhenoCHF
Relation TypeNo of annotated relations in narrative EHR reports
Causality 125
Negate 692


The relationship annotations were undertaken by two medical doctors. The quality and consistency of the annotations were verified through the calculation of inter-annotator agreement (IAA). We calculated IAA in terms of F-Score, and found that high levels of agreement were acheived (i.e., a macro-averaged F-Score of 0.91).