Conference paper Open Access

A Data-Driven Metric of Hardness for WSC Sentences

Nicos Isaak; Loizos Michael

JSON-LD ( Export

  "inLanguage": {
    "alternateName": "eng", 
    "@type": "Language", 
    "name": "English"
  "description": "<p>The Winograd Schema Challenge (WSC) | the task of resolving pronouns in certain sentences where shallow parsing techniques seem not to be directly applicable | has been proposed as an alternative to the Turing Test. According to Levesque, having access to a large corpus of text would likely not help much in the WSC. Among a number of attempts to tackle this challenge, one particular approach has demonstrated the plausibility of using commonsense knowledge automatically acquired from raw text in English Wikipedia. Here, we present the results of a large-scale experiment that shows how the performance of that particular automated approach varies with the availability of training material. We compare the results of this experiment with two studies: one from the literature that investigates how adult native speakers tackle the WSC, and one that we design and undertake to investigate how teenager non-native speakers tackle the WSC. We nd that the performance of the automated approach correlates positively with the performance of humans, suggesting that the performance of the particular automated approach could be used as a metric of hardness for WSC instances.</p>", 
  "license": "", 
  "creator": [
      "affiliation": "Open University of Cyprus, Nicosia, Cyprus", 
      "@type": "Person", 
      "name": "Nicos Isaak"
      "affiliation": "Open University of Cyprus, Nicosia, Cyprus & Research Center on Interactive Media, Smart Systems, and Emerging Technologies", 
      "@type": "Person", 
      "name": "Loizos Michael"
  "headline": "A Data-Driven Metric of Hardness for WSC Sentences", 
  "image": "", 
  "datePublished": "2018-09-17", 
  "url": "", 
  "version": "Accepted pre-print", 
  "@type": "ScholarlyArticle", 
  "keywords": [
    "Winograd Schema Challenge"
  "@context": "", 
  "identifier": "", 
  "@id": "", 
  "workFeatured": {
    "alternateName": "GCAI-2018", 
    "location": "Luxembourg City, Luxembourg", 
    "@type": "Event", 
    "name": "4th Global Conference on Artificial Intelligence"
  "name": "A Data-Driven Metric of Hardness for WSC Sentences"
Views 41
Downloads 35
Data volume 50.8 MB
Unique views 40
Unique downloads 35


Cite as