Conference paper Open Access

A Data-Driven Metric of Hardness for WSC Sentences

Nicos Isaak; Loizos Michael

Dublin Core Export

<?xml version='1.0' encoding='utf-8'?>
<oai_dc:dc xmlns:dc="" xmlns:oai_dc="" xmlns:xsi="" xsi:schemaLocation="">
  <dc:creator>Nicos Isaak</dc:creator>
  <dc:creator>Loizos Michael</dc:creator>
  <dc:description>The Winograd Schema Challenge (WSC) | the task of resolving pronouns in certain sentences where shallow parsing techniques seem not to be directly applicable | has been proposed as an alternative to the Turing Test. According to Levesque, having access to a large corpus of text would likely not help much in the WSC. Among a number of attempts to tackle this challenge, one particular approach has demonstrated the plausibility of using commonsense knowledge automatically acquired from raw text in English Wikipedia. Here, we present the results of a large-scale experiment that shows how the performance of that particular automated approach varies with the availability of training material. We compare the results of this experiment with two studies: one from the literature that investigates how adult native speakers tackle the WSC, and one that we design and undertake to investigate how teenager non-native speakers tackle the WSC. We nd that the performance of the automated approach correlates positively with the performance of humans, suggesting that the performance of the particular automated approach could be used as a metric of hardness for WSC instances.</dc:description>
  <dc:description>This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE – Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2)  and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.

©The authors</dc:description>
  <dc:publisher>Easy Chair</dc:publisher>
  <dc:subject>Winograd Schema Challenge</dc:subject>
  <dc:title>A Data-Driven Metric of Hardness for WSC Sentences</dc:title>
Views 41
Downloads 35
Data volume 50.8 MB
Unique views 40
Unique downloads 35


Cite as