No Golden Path - A Cautionary Tale of Quality and Bias

Narrative organization of information ties together storytelling in its many modalities. One archetypal expression of narratives we find in literary fiction. In this paper, we approach the elements of a successful narrative, and by extension storytelling, from the perspective of computational narratology. We are specifically interested in how to identify 'a good story'.

Quality assessment of literature is complicated by many factors. Literature is a complex linguistic phenomenon that conveys information indirectly, readers have different aesthetic preferences, and there is a lack of proper scientific instruments. One noisy, but ecologically valid measure is quantitative 'reader reviews.' On this account, a narrative is successful if readers rate it high. Such an 'extrinsic' success criterion is tempting because it is relatively easy to access, reflects readers' preferences in a natural setting, and its standardization appears trivial. A criterion that relies on reviews is however prone to several well-known biases, for instance, gender [1], ethnicity, and race [2], which point to fairness challenges in the classification of real-world data [3].  
Instead of merely relying of review annotation of the success of a story, we suggest paying attention to the inner structure of a story, termed the 'intrinsic success'. 

A recent theoretical paper has suggested that the affective coherence of a story, that is, the self-similarity of a sentiment story arc, functions as an index of a narrative's intrinsic success [4]. A complementary empirical study has shown that affective coherence can detect canonical literature [5].
(something is missing here)
While the use of computational narratology may seem compelling to minimize demographic disparities introduced by extrinsic success, it introduces less apparent and unknown biases. Genre, for instance, impacts a story arc and shows complex interactions with psychological propensities, aesthetic evaluation, and gender [1]. Socio-cultural norms may also play an important role in introducing unknown biases even at the methodological level.

In sum, there is no golden path to successful storytelling, that is, no single path that optimizes both quality and bias response. Instead, we see a multitude of possible trajectories, each of which implies different choices of known and unknown biases.

References

[1] S. Touileb, L. Øvrelid, E. Velldal,  Gender and sentiment, critics and authors: a dataset ofNorwegian book reviews,  in:  Proceedings of the Second Workshop on Gender Bias inNatural Language Processing, Association for Computational Linguistics, Barcelona, Spain(Online), 2020, pp. 125-138.

[2] P. Chong, Reading difference: How race and ethnicity function as tools for critical appraisal, Poetics 39 (2011) 64-84.

[3] T. Miconi, The impossibility of "fairness": a generalized impossibility result for decisions,2017.arXiv:1707.01195.

[4] Q. Hu, B. Liu, M. R. Thomsen, J. Gao, K. L. Nielbo,  Dynamic evolution of sentiments in Never Let Me Go: Insights from multifractal theory and its implications for literary analysis,Digital Scholarship in the Humanities 36 (2020) 322-332.

[5] Y. Bizzoni, T. Peura, M. R. Thomsen, K. L. Nielbo, Sentiment Dynamics of Success: Fractal Scaling of Story Arcs Predicts Reader Preferences, 2021. Proceedings of the First Workshop on Natural Language Processing for Digital Humanities (2021), 1-6.