Evaluation »

[ IE evaluation | KB evaluation ]

The KB evaluation is newly introduced in BioNLP-ST 2013.
It evaluates the performance of IE from a perspective of knowledge base construction.

For the evaluation,
  1. The final submission of each participating team will be converted into a SPARQL endpoint (an automatic process).
  2. Each SPARQL endpoint will be queried using a set of prepared biological queries.
  3. Each SPARQL endpoint will be evaluated by the richness and correctness of the answers it produces.

Note that the KB evaluation is experimental, and the organizers are open to accommodate any comment from participants. In this way, we want to make it a more "shared task".

Example SPARQL endpoint

As an example, the training and development data sets are accessible as a SPARQL endpoint at:
http://bionlp.dbcls.jp/sparql

Example SPARQL query

SELECT ?cd WHERE {
   ?e ao:hasTopic geo:Phosphorylation .
   ?c geo:causeOf ?e .
   ?c ao:context ?cs.
   ?cs ao:onSourceDocument ?cd .
}

("what documents talk about cause of phosphorylation?")

Coming soon

The KB evaluation is largely still under preparation. A more detail will come soon. So, please keep you eyes on the shared task home page. We will also announce of important updates through the BioNLP-ST mailing list.