NOTICE:

• Call For Papers for the special issue of Computational Intelligence.
• Online evaluation service for the test data set available.
• Shared task data sets released to public.
• Evaluation tools released to public.

Introduction

The BioNLP'09 Shared Task concerns the recognition of bio-molecular events (bio-events) that appear in biomedical literature.

The definition of bio-event is broadly described as a change on the state of a bio-molecule or bio-molecules, e.g. phosphorylation of IkB involves a change on the protein IkB.

The goal of the shared task is to provide common and consistent task definitions, datasets and evaluation for bio-IE systems based on rich semantics and a forum for the presentation of varying but focused efforts on their development.

Task definition

The BioNLP'09 Shared Task focuses on extraction of bio-events particularly on proteins or genes. (Proteins and gene are not distinguished.)

To concentrate efforts on the novel aspects of the extraction task, it is assumed that the protein recognition has been already performed, and the shared task begins with a gold standard set of proteins annotations.

The shared task is designed to address a semantically rich IE problem as a whole, but divided into three subtasks to allow separate evaluation of the performance for different aspects of the problem.

Task 1. Core event extraction (mandatory)

Participants are required to identify events concerning the given proteins. This task involves event trigger detection, event typing, and primary argument recognition.

Task 2. Event enrichment (optional)

Participants are required to find secondary argumentsents of events that further specify the event extracted by Task 1. This task involves the recognition of entities (other than proteins) and the assignment of these entities as event arguments.

Task 3. Negation and speculation recognition (optional)

Participants are required to find negations and speculations regarding events extracted by Task 1.

For detailed descriptions of the sub-tasks, please refer to the Examples section.

Data sets

The organizers provide human-curated reference material for the training and evaluation of participating systems. For training, a data set based on the publicly available portion of the GENIA corpus is be provided in a stand-off format. For evaluation, a held-out part of the same corpus is provided with the gold event annotation hidden. The goal for participating systems is to recreate the gold annotation based on the information induced from the training data.

The data sets are available for download at Downloads.

Other resources

To facilitate the development of systems making use of the wealth of available NLP tools, we provide simplified access to tools such as taggers and syntactic analyzers. In addition to the resources provided by the task organizers, participants are encouraged to develop their own and make use of other external tools and resources. Please see the tools page for more information.

Schedule

Procedure

1. System design period (1 month)
   • a sample data set is provided so that potential participants can start to prepare their IE systems.
2. System development period (1 month)
   • the final full training data set is provided for system training.
   • a sample test data set is provided.
   • a web site is made available to provide scoring for the sample test data.
3. Test period (1 week)
   • the final test set is provided.
   • the results from the participants are collected through the web site.

Important Dates

Dec. 8, 2008	First announcement
Dec. 15, 2009	Release of training data sample
Jan. 19	Release of full training data
Mar. 2	Release of the test data
Mar. 9	Deadline for submission of results
Mar. 10	Notification of the results of scoring
Mar. 23	Deadline for paper submission
Apr. 13	Notification of acceptance
Apr. 20	Deadline for camera ready paper submission
Jun. 5	BioNLP workshop special shared task sessions

Results

In total, 42 teams showed interest in the shared task and registered for participation. Among them, 24 teams submitted their final results. For the profile of the teams who allowed to reveal their identity, see Participants.

Since the tasks can be decomposed into various aspects, the evaluation is performed at various levels. For the detail of the evaluation methods, see Evaluation Methods.

The evaluation results are reported in the standard recall/precision/f-score evaluation scheme. For the final evaluation results, see Evaluation Results.

Workshop

All participants who submitted final results were invited to submit a manuscript describing their approach and result, and an overview paper was submitted by the organizers. Finally, 20 papers were accepted for 6 long, 7 short, and 7 poster presentations for the shared task session of the BioNLP'09 NAACL-HLT workshop.

For the program of the shared task session, see Program.

Organizers

Shared Task Chair :

• Jun'ichi Tsujii, University of Tokyo, University of Manchester, and National Centre for Text Mining

Organizing Committee:

• Jin-Dong Kim, University of Tokyo, on behalf of GENIA group
• Tomoko Ohta, University of Tokyo, on behalf of GENIA group
• Sampo Pyysalo, University of Tokyo, on behalf of BioInfer group
• Yoshinobu Kano, University of Tokyo, on behalf of U-Compare initiative

Program Committee:

• Andrew B. Clegg, University College London
• Nigel Collier, National Institute of Informatics
• Barry Haddow, University of Edinburgh
• Martin Krallinger, CNIO
• Ryan McDonald, Google
• Conrad Plake, Biotechnology Center of TU Dresden
• Thierry Poibeau, CNRS
• Burr Settles, University of Wisconsin-Madison
• Yoshimasa Tsuruoka, University of Manchester
• Hong Yu, University of Wisconsin-Milwaukee