Conference paper Open Access

A Data-Driven Metric of Hardness for WSC Sentences

Nicos Isaak; Loizos Michael

Citation Style Language JSON Export

{
"publisher": "Easy Chair",
"DOI": "10.29007/398z",
"container_title": "GCAI-2018. 4th Global Conference on Artificial Intelligence, EPiC Series in Computing, Volume 55",
"language": "eng",
"title": "A Data-Driven Metric of Hardness for WSC Sentences",
"issued": {
"date-parts": [
[
2018,
9,
17
]
]
},
"abstract": "<p>The Winograd Schema Challenge (WSC) | the task of resolving pronouns in certain sentences where shallow parsing techniques seem not to be directly applicable | has been proposed as an alternative to the Turing Test. According to Levesque, having access to a large corpus of text would likely not help much in the WSC. Among a number of attempts to tackle this challenge, one particular approach has demonstrated the plausibility of using commonsense knowledge automatically acquired from raw text in English Wikipedia. Here, we present the results of a large-scale experiment that shows how the performance of that particular automated approach varies with the availability of training material. We compare the results of this experiment with two studies: one from the literature that investigates how adult native speakers tackle the WSC, and one that we design and undertake to investigate how teenager non-native speakers tackle the WSC. We nd that the performance of the automated approach correlates positively with the performance of humans, suggesting that the performance of the particular automated approach could be used as a metric of hardness for WSC instances.</p>",
"author": [
{
"family": "Nicos Isaak"
},
{
"family": "Loizos Michael"
}
],
"page": "107-120",
"note": "This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE \u2013 Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2)  and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.\n\n\u00a9The authors",
"version": "Accepted pre-print",
"type": "paper-conference",
"id": "2671586"
}
18
17
views