Evaluation Set - Contributions Similarity in the Open Research Knowledge Graph

This evaluation set has been created for evaluating a content-based recommender system in the context of the Open Research Knowledge Graph (ORKG). The recommender system accepts structured ORKG contribution as input and recommends existing contributions in the ORKG semantically relevant to the given one.

The evaluation set is manually annotated based on the featured comparisons in the ORKG. In the course of this, it has been distinguished between homogeneous (those who are dissimilar in 2-3 properties) and heterogeneous (otherwise) instances. Multiple annotations have been obtained for the former and exactly one for the latter.

It has been also distinguished between "with_response" and "without_response" instances (50 instances for each). The former are those contributions for them the initial version of the contributions similarity service has found similarities and the latter are the opposite case.

This evaluation set has been created and applied on a modified version of the contributions similarity service in the context of this master's thesis. The modified version of the service has simplified the document representation of contributions that are stored in an ElasticSearch index by omitting redundant terms.

The evaluation set has the following schema:

{  
    "with_response": [
        {
            "contribution_id": "some_id",
            "comparison_id": "some_id",
            "comparison_label": "some_label",
            "contribution_label": "some_label",
            "paper": "some_id",
            "research_field": "some_id",
            "research_problems": [
                "some_id"
            ],
            "annotations": [
                "some_id of a similar contribution",
                ...
            ]
        },
        ...
      ],
    "without_response": [ ... ]
}