Published January 23, 2023 | Version v1
Working paper Open

Point Clouds. Scatterplots and Tables as User Interfaces of Artificial 'Intelligence'

Creators

Description

Scatterplots and tabular structures are the main graphical user interfaces for the diagrammatic depiction of large image data collections, which have been processed with visual recognition tools. This study discusses various media visualisations as case examples: ARTigo (LMU Munich), imgs.ai (UC Santa Barbara/Bildarchiv Foto Marburg), iArt (Universities of Munich and Hannover), Vikus Viewer (FH Potsdam), and X Degrees of Separation (Google). It further investigates how these projects employ visualisation algorithms like PCA, t-SNE and UMAP in relation to artificial weighted networks such as VGG19 or CLIP and provides practical proposals how to deal with the identified problems.

Files

5 Point Clouds. Scatterplots and Tables as User Interfaces of Artificial ‘Intelligence’_opt.pdf

Files (3.5 MB)

Additional details

References

  • Biederman, Irving. 1987. "Recognition-by-Components: A Theory of Human Image Understanding." Psychological Review 94 (2): 115–47. doi:10/c9dqh2
  • Bönisch, Dominik. 2021. "The Curator's Machine. Clustering von Musealen Sammlungsdaten Durch Annotieren Verdeckter Beziehungsmuster Zwischen Kunstwerken." Training the Archive – Working Paper, Aachen/Dortmund, May. doi:10.5281/ZENODO.4604880
  • Bowker, Geoffrey C. 2014. "The Theory/Data Thing Commentary." International Journal of Communication 8: 1795–99
  • Bowker, Geoffrey C., and Susan Leigh Star. 1999. Sorting Things Out – Classification and Its Consequences. Cambridge, MA: MIT Press
  • Brüggemann, Viktoria, Mark-Jan Bludau, Christopher Pietsch, and Marian Dörk. 2022. "Von Der Wolke Zum Pfad – Visuelle Und Assoziative Exploration Zweier Kultureller Sammlungen." In Zenodo. doi: 10.5281/ ZENODO.6327939
  • Bry, François, Clemens Schefels, and Corina Schemainda. 2018. "Eine Qualitative Analyse Der ARTigo-Annotationen: Serious Games, Games With A Purpose, Suchmaschine Für Kunstwerke, Kunstgeschichte, Human Computation, Semantische Tiefe." Edited by Piotr Kuroczyński, Peter Bell, and Lisa Dieckmann. Computing Art Reader. University Library Heidelberg, 96–114. doi: 10.11588/ARTHISTORICUM.413.C5771
  • Chun, Wendy Hui Kyong. 2021. Discriminating Data – Correlation, Neighborhoods, and the New Politics of Recogni-tion. Cambridge, MA: MIT Press
  • Correll, Michael. 2022. "Bad Ideas In Visualization." Medium. January 6. https://mcorrell.medium.com/bad-ideas-in-visualization-77d378148d35
  • D. Haggerty, Kevin, and Richard V. Ericson. 2000. "The Surveillant Assemblage." British Journal of Sociology 51 (4): 605–22. doi: 10.1080/00071310020015280
  • Deffebach, Nancy. 2015. María Izquierdo and Frida Kahlo: Challenging Visions in Modern Mexican Art. Latin American and Caribbean Arts and Culture. Austin, TX: University of Texas Press
  • Dobson, James E. 2019. Critical Digital Humanities – The Search for a Methodology. Topics in the Digital Humani-ties. Urbana, Illinois: University of Illinois Press
  • Few, Stephen. 2005. "Effectively Communicating Numbers – Selecting the Best Means and Manner of Display." Proclarity White Paper. https://www.perceptualedge.com/articles/Whitepapers/Communicating_Numbers.pdf
  • Few, Stephen 2012. Show Me the Numbers. Berkeley, CA: Analytics Press
  • Friendly, Michael, and Daniel Denis. 2005. "The Early Origins and Development of the Scatterplot." Journal of the History of the Behavioral Sciences 41 (2): 103–30. doi: 10.1002/jhbs.20078
  • Galloway, Alexander R. 2012. The Interface Effect. Cambridge: Polity Press
  • Geirhos, Robert, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019. "ImageNet-Trained CNNs Are Biased towards Texture – Increasing Shape Bias Improves Accuracy and Ro-bustness." ArXiv:1811.12231, January. http://arxiv.org/abs/1811.12231
  • Gitelman, Lisa. 2013. Raw Data Is an Oxymoron. Cambridge, MA: MIT Press
  • Glinka, Katrin, Christopher Pietsch, and Marian Dörk. 2017. "Von Sammlungsspezifischen Visualisierungen Zu Nachnutzbaren Werkzeugen." In Books of Abstracts, edited by Michael Stolz and Patrick Helling, 204–7. Bern. https://uclab.fh-potsdam.de/wp/wp-content/uploads/dhd2017-aus-programm.pdf
  • Hunger, Francis. 2021a. "'Why so Many Windows?' – Wie Die Bilddatensammlung ImageNet Die Automatisierte Bilderkennung Historischer Bilder Beeinflusst." Training the Archive – Working Paper 2, Aachen/Dortmund, June. doi: 10.5281/ZENODO.4742621
  • Hunger, Francis 2021b. "Curation and Its Statistical Automation by Means of Artificial 'Intelligence.'" Training the Archive – Working Paper 3, Aachen/Dortmund, November. doi: 10/gn8dn7
  • Hunger, Francis 2023. "Die Form Der Datenbank – Genealogien, Operationalitäten Und Praxeologien Relationaler Datenbanken in Ost Und West." Dissertation, Weimar: Bauhaus Universität Weimar [in preparation].
  • Hunger, Francis 2023. "'Why so Many Windows?' – How the ImageNet Image Database Influences Automated Image Recognition of Historical Images." International Journal for Digital Art History, [pre-print].
  • König, Dénes. 1936. Theorie Der Endlichen Und Unendlichen Graphen. Edited by Horst Sachs and H. Sachs. 1936 Reprint. Teubner-Archiv Zur Mathematik 6. Leipzig: Teubner
  • Krämer, Sybille. 2010. "Notationen Schemata Und Diagramme – »Räumlichkeit« als Darstellungsprinzip." In Notati-onen Und Choreographisches Denken., edited by Gabriele Brandstetter, Franck Hofmann, and Kirsten Maar, 29–45. Freiburg/Berlin/Wien: Rombach Verlag
  • Krämer, Sybille, and Horst Bredekamp. 2003. "Kultur, Technik, Kulturtechnik – Wider Die Diskursivierung Der Kultur." In Bild, Schrift, Zahl., edited by Sybille Krämer and Horst Bredekamp, 11–22. Kulturtechnik. München: Fink
  • Kubovy, Michael, and Martin van den Berg. 2008. "The Whole Is Equal to the Sum of Its Parts – A Probabilistic Model of Grouping by Proximity and Similarity in Regular Patterns." Psychological Review 115 (1): 131–54. doi: 10/b2vw3s.
  • Latour, Bruno. 1987. Science in Action – How to Follow Scientists and Engineers through Society. Cambridge, MA: Harvard University Press
  • Manovich, Lev. 2020. Cultural Analytics. Cambridge, MA: The MIT Press
  • McInnes, Leland, John Healy, and James Melville. 2020. "UMAP – Uniform Manifold Approximation and Projection for Dimension Reduction." ArXiv:1802.03426 [Cs, Stat], September. http://arxiv.org/abs/1802.03426
  • Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et. al. 2021. "Learning Transferable Visual Models From Natural Language Supervision." arXiv. http://arxiv.org/ abs/2103.00020
  • Russell, Henry Norris. 1914. "Relations between the Spectra and Other Characteristics of Stars." In Popular Astron-omy, 22:275–94. https://ia802300.us.archive.org/16/items/sim_popular-astronomy_1914-05_22_5/sim_popular-astronomy_1914-05_22_5.pdf
  • Sarikaya, Alper, and Michael Gleicher. 2018. "Scatterplots – Tasks, Data, and Designs." IEEE Transactions on Visua-lization and Computer Graphics 24 (1): 402–12. doi: 10.1109/TVCG.2017.2744184
  • Schneider, Stefanie, and Hubertus Kohle. 2021. "Bilderlesemaschinen – Einsatzszenarien Maschineller Lernverfah-ren Im Musealen Raum." Museum and the Internet (MAI) 2021 Tagung, May 28. https://www.youtube.com/ watch?v=lsSpzjtS-rM
  • Simonyan, Karen, and Andrew Zisserman. 2015. "Very Deep Convolutional Networks for Large-Scale Image Recognition." ArXiv:1409.1556 [Cs], April. http://arxiv.org/abs/1409.1556
  • Springstein, Matthias. 2021. "AW: IArt Fragen." Email to the Author
  • Springstein, Matthias, Stefanie Schneider, Javad Rahnama, Eyke Hüllermeier, Hubertus Kohle, and Ralph Ewerth. 2021. "IART – A Search Engine for Art-Historical Images to Support Research in the Humanities." ArXiv:2108.01542 [Cs], August. http://arxiv.org/abs/2108.01542
  • Taylor, Frederick Winslow. 1903. Shop Management. London, New York, NY: Harper & Brothers
  • Toshev, Alexander, and Christian Szegedy. 2014. "DeepPose – Human Pose Estimation via Deep Neural Networks." 2014 IEEE Conference on Computer Vision and Pattern Recognition, June, 1653–60. doi: 10/ggtmv3
  • Tufte, Edward R. 1983. The Visual Display of Quantitative Information. Cheshire: Graphics Press
  • Wilke, C. 2019. Fundamentals of Data Visualization – A Primer on Making Informative and Compelling Figures. Sebastopol, CA: O'Reilly Media
  • Zhaoping, Li. 2019. "A New Framework for Understanding Vision from the Perspective of the Primary Visual Cortex." Current Opinion in Neurobiology 58 (October): 1–10. doi: 10/ghndbt