Citation review of Human-Machine Collaboration in Virtual Reality for Adaptive Production Engineering by de Giorgio et al.

This article argues that despite a citation review is a rarely used research tool, this can be very useful to assess the impact of new research topics, both from the future research direction and the bibliometric perspectives. An explorative study is presented around the research area marked as Industry 4.0 with the conference paper mentioned in the title of this citation review. Even though the given reference paper is relatively recent, there are already twenty-seven citations listed among three different scholar databases. These are Google Scholar, ResearchGate and Semantic Scholar. In light of this, the article provides a bibliometric confirmation and analysis for the progression of the line of research adopted by de Giorgio et al. in the exploration of non-traditional methods using virtual reality technology and human-robot collaboration for adaptive applications in Industry 4.0. Furthermore, it represents a model for the authors’ self -development and an example of an unconventional approach to scientific work that may help improve related bibliometric research and scholar database strategies to index new articles and topics in the future.


Background
The importance of citation review papers is of growing importance within the context of Industry 4.0, a notion that is slowly gaining ground. As the topic of study becomes ever more multi-disciplinary, the possibility for a research team to critically appraise all of its own developments tends to become incrementally difficult, if not impossible. The majority of research teams dealing with Industry 4.0 are either primarily production engineers or mainly computer & control experts. Ideally, however, one would prefer the developments to be carried out by multi-disciplinary teams in order to capture the essential details required for successful industrial applications. Due to this lack of diversity in the research teams, the citation review articles gain importance as critical appraisals of someone's own work can be gathered from all disciplines. The work described by de Giorgio et al. (2017), and analyzed in terms of its citations, is a clear example of such multidisciplinarity as it spans from pure computer science and virtual reality to production engineering and collaborative robots (cobots), with educational and control aspects included. Considering these inherent limitations, such as the common difficulty for mechanical engineers to abstract architectural requirements, or general engineers' limitations in pedagogical approaches, the need to critically appraise the research becomes more evident.
Interactive 3D Environment for Learning Real Life Cooking Tasks" (Gao et al. 2019b) from 2019 and "A Systematic Literature Review: Human Roles, Competencies And Skills In Industry 4.0" (Janis and Alias 2018); the preprint is "Augmented Perception: Can virtual environments have an impact on well-being and work effectiveness?" (Marszalek and Volz 2018).
RG is missing nine articles that are reported by GS. Among those, one is the journal article "Strategic systematic for software development in industry 4.0" (Oliveira Valério et al. 2020). Four are conference papers (Arntz and Eimler 2020;Etzi et al. 2019;Metzner, Utsch, et al. 2020), specifically "Experiencing AI in VR: A Qualitative Study on Designing a Human-Machine Collaboration Scenario" (Arntz and Eimler 2020), "Use of Virtual Reality in Product Development by Distributed Teams" (H. P. Balzerkiewitz and Stechert 2020), "A system for human-in-the-loop simulation of industrial collaborative robot applications" (Metzner, Utsch, et al. 2020) and "Using virtual reality to test human-robot interaction during a collaborative task" (Etzi et al. 2019). Four are theses (Gong 2019;Gustavsson 2020; Lloyd Reyson Robles 2020; Tolman 2018).
SS is missing 9 citations from RG and GS. In particular, the Portuguese journal article "Simulação como Tecnologia Habilitadora da Indústria 4.0: Uma Revisão da Literatura" (Randon and Cecconello 2019); three conference papers (Arntz and Eimler 2020; H. P. Etzi et al. 2019) that are "Using virtual reality to test human-robot interaction during a collaborative task" (Etzi et al. 2019), "Experiencing AI in VR: A Qualitative Study on Designing a Human-Machine Collaboration Scenario" (Arntz and Eimler 2020)
A keyword analysis shows ninety-nine keywords associated with the citing work. The most common ones are Virtual Reality (twelve occurrences), Simulation (five occurrences) and Human-Robot-Collaboration (four occurrences). For a complete overview of the keyword count, see table 1. A treemap shows graphically the keyword distribution in figure 3.  The citing work comes from 13 countries with the distribution shown in Fig. 4. In particular, six citations come from USA, five from Germany, two from Brazil, Norway, Italy and Sweden. The rest of the countries give one citation each.

Summary of findings
The related work section of Peruzzini et al. (2020) lists de Giorgio et al. (2017) as those who "showed the potentiality of virtual reality (VR) in human-robot collaboration, using game engines as simulation software to control AR/VR hardware and industrial machines". Further in the article, a table named "Comparison between different VM-based approaches" lists de Giorgio et al. (2017) as an application of collaborative robotics with a level of realism of score 3 out of 4, using Unity as VR software and HTC Vive with VR Controllers as hardware. The table also explains that de Giorgio et al. (2017) performed a qualitative interaction evaluation over a VR application with level of interaction that is "360° visual" other than "tactile with feedback". These results are of some importance as they may indicate in which area there may exist a need to improve or further elaborate.
In a section called "basics",  state that virtual reality is often characterized by five characteristics that are stated by several authors, including de Giorgio et al. (2017), these are a virtual world, immersion, feedback, interactivity and participants. This entails that the basic foundations of the cited article does find positive theoretical backing from other authors, which offers grounds for validation.
In the introduction, Ojstersek et al. (2019) use de Giorgio et al. to claim that "researchers relate the VR model to the real environment in cooperative workplaces, which represents a rapidly growing research field with broad applicative solutions". Once again, this enhances the basic validity of the original work.
In the related work section, Metzner et al. (2020)  In the introduction, Arntz and Eimler (2020) use de Giorgio et al. (2017) to claim that Virtual Reality (VR) delivers a tool to validate concepts for the production processes of the future, before that they can become reality.
In the introduction, Lloyd Reyson Robles (2020) 2017) as a human-robot platform in which the user "wears a head-mounted display (HDM; HTC Vive™) and moves a robotic arm (ABB IRB 120) through a controller to perform an assembly task".
In a section named "a brief overview about virtual reality", Balzerkiewitz and Stechert (2020) mention de Giorgio et al. (2017) among other works to state that virtual reality is often characterized by five characteristics: a virtual world, immersion, feedback, interactivity and participants. Later in the article, they further observe through de Giorgio et al. (2017), among a few works, that most CAD metadata, e.g. constraints, color, and material, is lost during the CAD exchanging process from the virtual reality world into classical CAD software.
In a section named "state of the art in human-robot collaboration system simulation", Metzner et al. (2020) mention de Giorgio et al. as those who "integrate an industrial robot kinematic into the gaming engine Unity™ to enable VR visualization and virtual control of the robot arm".
Oliveira Valério et al. (2020) cite de Giorgio et al. (2017) in a section named "software development and industry 4.0": they name this work, among others, as those identifying the risks/challenges of software integration and new technologies. These are "the lack of systems compatibility, the labor replacement, the impact of consumer relations, the structure of big data, the security and robustness of information systems (legislation), the expansion of Internet services and communication, and the diffusion of the new technological scenario for students and researchers". The article further uses de Giorgio et al. (2017) to justify the claim that "possible risks/challenges should be discussed at the stage concerning the impact of extremely exclusive consumer and product relationships (Determine how customer dissatisfaction and possible discards should be regulated)." In the related work of Gong's thesis (Gong 2019), de Giorgio et al. (2017 is presented as an example of research in human-robot interaction with virtual reality. Note that this thesis contains the material used to produce the preprint "VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks" (Gao et al. 2019b) and the conference paper "VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning" (Gao et al. 2019a).
In a section on human-robot collaboration, subsection about virtual commissioning and virtual reality, Gustavsson (2020) cites de Giorgio et al. (2017) among "recent studies that have shown great potential in virtual training and testing using VR, which potentially could include the human in the commissioning process using a virtual plant". Note that this PhD thesis links to a paper named "Evaluation of Human-Robot Interaction for Assembly Manufacturing in Virtual Reality" authored by Patrik Gustavsson, Magnus Holm and Anna Syberfeldt, sent to Elsevier for publication. This paper cites de Giorgio et al. (2017) in the related work section, as a research concluding that "virtual reality for human-robot collaboration provides the possibilities for the operator to learn how to perform manufacturing processes."

Citation clusters
Among the most citing authors are Ran Gong, Xiaofeng Gao and their colleagues with the VRKitchen series (Gao et al. 2019a(Gao et al. , 2019bGong 2019). They start in 2019 with Ran Gong's thesis "VRKitchen: A 3D Dynamic Interactive Environment for General Computer Vision Research" (Gong 2019), then publish in the same year a preprint "VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning" (Gao et al. 2019a) and a conference article "VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks" (Gao et al. 2019b).
Beibei Shu and colleagues in 2018 and 2019 publish two conference papers, namely "Human-robot collaboration: Task sharing through Virtual Reality" (Shu et al. 2018) and "Collaboration with high-payload industrial robots: Simulation for safety" (Shu and Sziebig 2019).
Hans Patrick Balzerkiewitz and colleagues in 2020 publish two conference papers, namely "The evolution of virtual reality towards the usage in early design phases" (H.-P. Balzerkiewitz and Stechert 2020) and "Use of Virtual Reality in Product Development by Distributed Teams" (H. P. . Maximilian Metzner and colleagues in 2020 publish two conference papers, namely "A system for humanin-the-loop simulation of industrial collaborative robot applications" (Metzner, Utsch, et al. 2020) and "Intuitive, VR-and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models" (Metzner, Krieg, et al. 2020).

Citation errors
While all the citations are correctly addressing the content of the cited paper, the author's name used for the citation is often wrong. In fact, 45% of the citations are wrongly spelled. Half of them are simply using a capitalized "De" instead of an uncapitalized "de" for the last name, which appears as " Despite the many errors, the scholar databases seem in most cases able to catch the reference to the paper. However, there are a few missing documents in one search engine or another and the misspelling of the citation key could play a role in this problem. Sadly, in this citation review there are not enough missing samples to investigate such hypothesis. Future work should include such an investigation on a large source of misspelled citations.

Document type errors
Many documents are reported by the scholar database with the wrong document type attribution, i.e. journal article, conference paper, chapter, etc. In particular, on ResearchGate six conference papers are recorded as journal article (two occurrences), as chapter (three occurrences) or as data (one occurrence). One document on Semantic Scholar is named after the whole conference title in spite of the conference paper name (this has been removed from the citation review for obvious reasons); the conference paper was also reported in the same database as a separate document and included in this citation review. In one case, on RG the conference paper had the wrong publication year. Google Scholar and Semantic Scholar do not explicitly define the document type, thus it is not possible to perform the same document type analysis done for RG. However, the citing tool offered by the original publisher (in particular Springer) or the citation database extractor tend to wrongly mark conference paper types as a chapter types.

Conclusions
From the summary of findings, it is possible to count seven citations among the introductions. Of those, one is particularly interesting as Anik et al. (2020) open their paper citing de Giorgio et al. as an example of application that is possible due to recent breakthroughs in VR technology, placing de Giorgio et al. at the top of its scope. A higher number of citations comes from the related work sections (eleven occurrences) and literature review sections (one occurrence), which mainly attests the similarity of scope between de Giorgio et al. and the citing articles. Equally great, with ten citations, is the number of mentions in the central sections of the citing articles, which provides a reasonable proof that de Giorgio et al. paves the way to new findings in the application of virtual reality in human-robot collaboration for adaptive production engineering.
From the work of Randon and Cecconello (2019) it results that the choice of publication venue, i.e. Procedia Manufacturing as part of the conference proceedings, was effective in delivering this kind of content to a broad audience. They point out that applications of simulation for Industry 4.0 are mostly published and cited from Procedia Manufacturing instead of other important journals in the field of manufacturing.
On the other hand, only 18% of the citations come from journal articles, while 59% of them come from other conference papers. Another 15% come from theses and 7% come from preprints that do not specify what type of article will be published.
Particularly appreciated is the reviewed definition of the five main characteristics of VR applications: a virtual world, immersion, feedback, interactivity and participants. Also the valid industrial scenario presented in the paper that allows to evaluate advantages and risks of applying VR in human-robot collaborative applications for Industry 4.0.
None of the citing papers tries to undermine or criticize the work presented by de Giorgio et al. (2017), which is expected as it relates to the low percentage of this kind of citations. Only one comment addresses a limitation regarding the test of the industrial scenario with only one robot model. Indeed, a perfect example of future work to develop. As named earlier, these citations, once put into their correct context, offer both validation data and potential areas of improvement. In other words, the citation review may act as an external tutorship in which both similar and adjacent disciplines can analyze the results within specific contexts.
One limitation of this citation review is the young age of the publication in exam and the relative low number of citations that can be collected from it. Despite that, having few citations allows to track them more precisely and it shows that scholar databases are constantly missing citations and each one of them indexes only a limited subset of all the existing scientific publications. A huge result is to count 27 citations over a maximum of 23 spotted by Google Scholar, and 19 different ones spotted by ResearchGate and Semantic Scholar.
A quick overview of the keyword treemap presented in Fig. 3 reveals what was mentioned earlier: there are very few keywords that stem from Industry 4.0-related work. This indicates that although this topic should be focusing on the IT and software enhancement of industry, the interest for virtual reality applications in industry is still coming from its conceptual developers. This should be highlighted since the integration of new technologies in a mature industrial environment is notoriously difficult.
Lastly, a reasonable speculation that can be done in conclusion of this work is that, eventually, the fact that this very article will cite non-indexed scientific work could lead to the scholar databases to use the citations hereby provided to track and index them. In fact, citations are the basis of automated document retrieval and indexing algorithms. Time will help to verify this hypothesis.