Event Annotation

Annotation scheme

Four types of events are annotated to encode detailed, structured information about the effects of drugs. Definitions and possible participants are shown in Table 2, together with some examples.

Table 1. Event Types
Event TypeDescriptionPossible Participants
Adverse_effect (AE) A pharmacological substance or combination/interaction between pharmacological substances has an effect on the body that is considered to be undesirable. More specifically, the substance/combination causes a disorder to manifest itself, or to become worse. has_agent: Pharmacological substance or combination/interaction of substances responsible for the adverse effect.
affects: Disorder resulting from or worsened by administration of the Agent
has_subject: Individual or group (human or otherwise) in which the drug effect is specified to occur

Potential_therapeutic_effect (PTE) A pharmacological substance or combination/interaction of pharmacological substances is being administered, with the intention of having a therapeutic effect. has_agent: Pharmacological substance or combination/interaction of substances responsible for the therapeutic effect
affects: Disorder improved or cured by administration of the Agent
has_subject: Individual or group (human or otherwise) in which the therapeutic effect is specified to occur.
Combination A specification that two or more pharmacological substances are being used at the same time (e.g., they have been co-administered). has_participant: A pharmacological substance being combined/co-administered
DDI A specific mention that there is an interaction between two or more pharmacological substances. has_participant: A pharmacological substance involved in the interaction
has_subject: Individual or group (human or otherwise) in which the DDI is specified to occur.

Table 2 outlines the conditions (in terms of the presence of participants) that must hold for each event type to be annotated, according to what is considered to be the minimal useful information; events are not annotated if these conditions are not met.

Table 2. Conditions for event annotation
Event TypeConditions
Adverse_effect (AE)
  • At least ONE has_agent participant
Potential_Therapeutic_Effect (PTE)
  • At least ONE has_agent participant
  • At least ONE affects participant OR at least ONE has_subject participant

Combination
  • At least TWO has_participant participants

DDI
  • At least TWO has_participant participants

Event annotation statistics

All mentions of the event types shown in Table 1 were annotated in all abstracts in the corpus. The total number of event annotations are shown in Table 3, together with the unique number of triggers.

Table 3. Statistics of event annotations
Event TypeNumber of event annotationsNumber of unique triggers
Adverse_effect (AE) 1375 309
Potential_therapeutic_effect(PTE) 893195
Combination 56081
DDI 33812

Table 4 shows the total number of the diferent types of event participants annotated.

Table 4. Statistics of event participants
Participant typeNumber of annotations
has_agent 2424
affects 2308
has_subject909
has_participant1882

Agreement

The event annotations were undertaken by annotators with domain expertise. The quality and consistency of the annotations were verified through the calculation of inter-annotator agreement (IAA) on one quarter of the complete corpus (i.e., 150 abstracts). We calculated IAA in terms of F-Score. We calculated two types of agreement rates that correspond to the different stages of event annotation, i.e.:

  1. Event identification (i.e., the extent to which the annotators agree that particular events are described in a given sentence)
  2. Participant identification (i.e., the extent to which the annotators agree on which participants to annotate for events whose existence they agree upon).

Event identification agreement

Event identification involves selecting an appropriate trigger word or phrase for the event. We have defined strict and relaxed matching criteria for evaluating event identification agreement; the results are shownin Table 5

  • Strict criterion - requires that the event trigger spans chosen by both annotators include some degree of overlap.
  • Relaxed criterion - Given the potential difficulty in selecting an appropriate event trigger, and the fact that there may be multiple possible candidates in a sentence, we assume that event triggers refer to the same event if any of the following conditions hold:
    • Triggers overlap AND they are assigned the same event type; OR
    • Triggers occur within 20 characters of each other AND they are assigned the same event type; OR
    • Triggers occur in the same sentence AND they are assigned the same event type AND they share at least one "core" participant (i.e., at least one has_agent, affects or has_participant participant must be the same in the matched events).

Table 5. Inter-annotator agreement rates for event identification(F-score)
Agreement TypeStrict AgreementRelaxed Agreement
PTE 60.2 72.5
AE63.980.3
Combination 65.374.9
DDI 86.587.1
TOTAL 65.078.5

Participant identification agreement

Table 6 shows the extent to which the annotators identified the same participants with the same semantic roles for events whose existence they agreed upon (according to the relaxed criteria introduced above). Event participants chosen by the two annotators are considered to match if:

  • their text spans overlap AND
  • they are assigned the same semantic role

Table 5. Inter-annotator agreement rates for event participants (F-score)
Agreement TypeAgreement
has_participant 96.1
has_agent86.2
affects 90.1
has_subjct 73.0
TOTAL 88.2