Published September 17, 2018 | Version Accepted pre-print
Conference paper Open

Using the Winograd Schema Challenge as a CAPTCHA

  • 1. Open University of Cyprus, Nicosia, Cyprus
  • 2. Open University of Cyprus, Nicosia, Cyprus & Research Center on Interactive Media, Smart Systems, and Emerging Technologies


CAPTCHAs have established themselves as a standard technology to condently distinguish humans from bots. Beyond the typical use for security reasons, CAPTCHAs have helped promote AI research in challenge tasks such as image classication and optical
character recognition. It is, therefore, natural to consider what other challenge tasks for AI could serve a role in CAPTCHAs. The Winograd Schema Challenge (WSC), a certain form of hard pronoun resolution tasks, was proposed by Levesque as such a challenge task to promote research in AI. Based on current reports in the literature, the WSC remains a challenging task for bots, and is, therefore, a candidate to serve as a form of CAPTCHA. In this work we investigate whether this a priori appropriateness of the WSC as a form of CAPTCHA can be justied in terms of its acceptability by the human users in relation to existing CAPTCHA tasks. Our empirical study involved a total of 329 students, aged between 11 and 15, and showed that the WSC is generally faster and easier to solve than, and equally entertaining with, the most typical existing CAPTCHA tasks.


This work has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 739578 and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development. ©The authors



Files (722.5 kB)

Additional details


RISE – Research Center on Interactive Media, Smart System and Emerging Technologies 739578
European Commission


  • Evan Ackerman. Winograd Schema Challenge Results: AI Common Sense Still a Problem, for Now. Spectrum, 2016.
  • Pascal Amsili and Olga Seminck. A Google-Proof Collection of French Winograd Schemas. In Second Workshop on Coreference Resolution beyond OntoNotes, page 24, 2017.
  • Sandhya Tarar Anvesh Sinha. Review Paper on Dierent CAPTCHA Techniques. IJCST Vol. 7, 2016.
  • Marios Belk, Panagiotis Germanakos, Christos Fidas, Andreas Holzinger, and George Samaras. Towards the Personalization of CAPTCHA Mechanisms Based on Individual Dierences in Cog- nitive Processing. In Human Factors in Computing and Informatics, pages 409{426. Springer, 2013.
  • David Bender. Establishing a Human Baseline for the Winograd Schema Challenge. In MAICS, pages 39{45, 2015.
  • Kevin Bock, Daven Patel, George Hughey, and Dave Levin. unCaptcha: A Low-Resource Defeat of reCaptcha's Audio Challenge. In Proceedings of the 11th USENIX Conference on Oensive Technologies, pages 7{7. USENIX Association, 2017.
  • Ernest Davis, Leora Morgenstern, and Charles Ortiz. Human tests of materials for the Winograd Schema Challenge 2016. 2016.
  • Jeremy Elson, John JD Douceur, Jon Howell, and Jared Saul. Asirra: A CAPTCHA that Exploits Interest-Aligned Manual Image Categorization. 2007.
  • Christos A Fidas, Artemios G Voyiatzis, and Nikolaos M Avouris. On the necessity of user-friendly CAPTCHA. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2623{2626. ACM, 2011.
  • Walid Khalifa Abdullah Hasan. A Survey of Current Research on Captcha. International Journal of Computer Science and Engineering Survey (IJCSES), 7(3):141{157, 2016.
  • Carlos Javier Hernandez-Castro and Arturo Ribagorda. Pitfalls in CAPTCHA design and imple- mentation: The Math CAPTCHA, a case study. computers & security, 29(1):141{157, 2010.
  • Nicos Isaak and Loizos Michael. Tackling the Winograd Schema Challenge Through Machine Logical Inferences. In David Pearce and Helena Soa Pinto, editors, STAIRS, volume 284 of Frontiers in Articial Intelligence and Applications, pages 75{86. IOS Press, 2016.
  • Nicos Isaak and Loizos Michael. A Data-Driven Metric of Hardness for WSC Sentences. In Proceedings of the 4th Global Conference on Articial Intelligence (GCAI 2018). EasyChair, 2018.
  • Hector J. Levesque. The Winograd Schema Challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, number SS-11-06. American Association for Articial Intelligence, 2011.
  • Vu Duc Nguyen, Yang-Wai Chow, and Willy Susilo. On the security of text-based 3D CAPTCHAs. Computers & Security, 45:84{99, 2014.
  • Haoruo Peng, Daniel Khashabi, and Dan Roth. Solving Hard Coreference Problems. Urbana, 51:61801, 2015.
  • Altaf Rahman and Vincent Ng. Resolving Complex Cases of Denite Pronouns: The Winograd Schema Challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL '12, pages 777{789, Stroudsburg, PA, USA, 2012. Association for Computational Linguistics.
  • Adam Richard-Bollans, L Gomez Alvarez, and Anthony G Cohn. The Role of Pragmatics in Solving the Winograd Schema Challenge. In Proceedings of 13th International Symposium on Commonsense Reasoning (Commonsense-2017). CEUR Workshop Proceedings, 2017.
  • By Tara Seals. unCAPTCHA Defeats Google CAPTCHA, 2017. [Online; accessed August-2018].
  • Arpit Sharma, Nguyen H Vo, Somak Aditya, and Chitta Baral. Towards Addressing the Winograd Schema Challenge - Building and Using a Semantic Parser and a Knowledge Hunting Module. In Proceedings of the Twenty-Fourth International Joint Conference on Articial Intelligence, IJCAI, pages 25{31, 2015.
  • Suphannee Sivakorn, Jason Polakis, and Angelos D Keromytis. I'm not a human: Breaking the Google reCAPTCHA. Black Hat,(i), pages 1{12, 2016.
  • Google no Captcha + INVISIBLE reCaptcha { First Experience Results Review, 2017. [Online; accessed August-2018].
  • Christiana Themistocleous. Written Cypriot Greek in online chat: Usage and attitudes. In Pro- ceedings of the 8th International Conference on Greek Linguistics, volume 30, pages 473{488. University of Ioannina Ioannina, 2009.
  • By Liam Tung. Google algorithm busts CAPTCHA with 99.8 percent accuracy, 2017. [Online; accessed August-2018].
  • By Rob Verger. Google just made the Internet a tiny bit less annoying, 2017. [Online; accessed August-2018].
  • Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. CAPTCHA: Using Hard AI Problems For Security. In International Conference on the Theory and Applications of Cryp- tographic Techniques, pages 294{311. Springer, 2003.
  • Jeff Yan and Ahmad Salah El Ahmad. Breaking Visual CAPTCHAs with Naive Pattern Recog- nition Algorithms. In Computer Security Applications Conference, 2007. ACSAC 2007. Twenty- Third Annual, pages 279{291. IEEE, 2007.
  • Jeff Yan and Ahmad Salah El Ahmad. Usability of CAPTCHAs or usability issues in CAPTCHA design. In Proceedings of the 4th symposium on Usable privacy and security, pages 44{52. ACM, 2008.