Can Robots Impact Human Comfortability During a Live Interview?

Interaction among humans does not always proceed without errors; situations might happen in which a wrong word or attitude can cause the partner to feel uneasy. However, humans are often very sensitive to these interaction failures and may be able to fix them. Our research aims to endow robots with the same skill. Thus the first step, presented in this short paper, investigates to what extent a humanoid robot can impact someone's Comfortability in a realistic setting. To capture natural reactions, a set of real interviews performed by the humanoid robot iCub (acting as the interviewer) were organized. The interviews were designed in collaboration with a journalist from the press office of our institution and are meant to appear on the official institutional online magazine. The dialogue along with fluent human-like robotic actions were chosen not only to gather information about the participants' personal interests and professional career, necessary for the magazine column, but also to influence their Comfortability. Once the experiment is completed, the participants' self-report and spontaneous reactions (physical and physiological cues) will be explored to tackle the way people's Comfortability may be manifested through non-verbal cues, and the way it may be impacted by the humanoid robot.


INTRODUCTION
Discovering the capabilities that are key to pleasantly interact with people, might be the initial step to enhance current humanoid robots as social partners. Identifying others' emotional and affective states is fundamental to maintain natural and effective interactions [5]. However, modelling all possible individual emotions is still a challenge; from the way they might be manifested [2], to the way they should be interpreted by artificial intelligent systems [10].
For this reason we decided to explore Comfortability, which was previously introduced in [11] as a single representation of people's feelings during an interaction. It is placed in a Extremely Uncomfortable to Extremely Comfortable uni-dimensional scale, and defined as "(disapproving of or approving of) the situation that arises as a result of an interaction which influences one's own desire of maintaining or withdrawing from it". Redondo, et al. [11] explored already the impact of a humanoid robot on someone's Comfortability. Concretely, they exposed participants with prerecorded videos of the humanoid robot iCub [9] acting as an interviewer in a virtual and imaginary interaction. The participants reported that their Comfortability would have been affected by robotic actions, had they been part of such situation.
Given the previous study missed an in-presence interaction, this paper proposes an experiment based on the same scenario but involving participants in a live interaction with the robot. Hence, in collaboration with the press office, a set of real interviews between iCub (acting as the interviewer) and researchers from our institution (being the interviewees) were designed. Of course, the interviews are real as the recorded materials will be used by the journalist to present the interviewees' research on the online institutional magazine. Considering the ecological setting, we intend to collect natural and spontaneous reactions to discover: 1) to what extent a humanoid robot can impact human Comfortability, and 2) which are the physical and physiological cues manifested when experiencing specific Comfortability levels.

STATE OF THE ART
Although Comfortability is constantly present in our lives, we barely found any research focused on this concept. Conversely, stress [12], awkwardness [8] and similar feelings have been deeply addressed.

Late-Breaking Report
HRI '21 Companion, March 8-11, 2021, Boulder, CO, USA Nevertheless, these internal states do not provide the same information as Comfortability does. For example, two persons might be playing a competitive game and feel highly stressed and/or nervous as their aim is to perform well and win; but at the same time, they might be comfortable (with a high positive Comfortability level) because they want to keep with the ongoing interaction and they are enjoying the current moment.

Previous HRI studies on Comfortability
Even though people might feel uncomfortable in certain interactions, it is not clear to what extent humanoid robots are able to provoke similar feelings when interacting with them. As a consequence, the field of Human-Robot Interaction (HRI) has started to approach the concept of Comfortability, although it is often referred to under the name Comfort.
Koay et al. [7] developed a handheld "Comfort Level Device" and made participants report their own Comfort while performing a task in a simulated living room scenario in the presence of the Peo-pleBot robot. The subject had to search some books and write their titles on a whiteboard while the robot was moving around. They found that the situations in which the robot was moving behind the subjects, blocking them or colliding with their path were the ones reported as more uncomfortable. Also, Ball et al. [1] studied people's Comfort regarding an approaching robot. Specifically, two persons engaged in a collaborative task (solving a jigsaw puzzle) were approached by the Adept Pioneer 3DX robot from 8 different angles. They found that the approaches from all front directions were reported as more comfortable than those by the shared rear direction. Recently, Chatterji et al. [4] studied people's likeability, understandability and comfortability when interacting with a robot. They created some video clips where different robots (Atlas, Cozmo, Roomba, Fetch, Jaco, Jibo, Kuri, Moxi, Nao, Pepper and Sawyer) were presented in three conditions: 'emitting sound', 'voice' or a 'mix of both'. They discovered that as the robot became more anthropomorphic and/or social, 'voice' only was preferred for the three attributes under study. And, Sicat et al. [13] explored whether social robots should be programmed to obey humans or act as their leaders. To answer this question, they implemented the "Mirror game" including a human and the Baxter robot. In the first stage, the human started by leading the movement which the robot should imitate, until the experimenter decided to change to the next phase (the robot leading). To make that decision, the experimenter applied their own judgement assessing if the participant was comfortable enough; which clearly highlights the necessity of comprehending Comfortability in all interactive agents.

Motivation
Considering the mentioned papers, it can be seen that Comfortability is present in HRI, however most of the time a definition and formalization is clearly demanded. That is to say, when the researchers and/or participants were addressing it, no definition was provided, nor the subsequent reactions were studied. For this reasons, our goal after completing the data collection of this experiment is to provide a deep analysis of this internal state, build a database with the associated reactions, and code an artificial intelligence (AI) capable of identifying people's Comfortability.

Cover Story
As mentioned in the Introduction, this paper builds upon the study by Redondo et al. [11] where participants were requested (through an online questionnaire) to imagine being interviewed by a reporter while specific actions were presented to them. Depending on the experimental condition, the actions were shown through sentences (Narrative Context) or videos of the robot iCub (Visual Context).
The robotic actions used in the current real-life experiment were inspired by those proposed in the Visual Context condition [11]. To maximize the immersion, participants were recruited by the institutional press office IIT OpenTalk department and informed that their interview will be published on the online institutional magazine IO IIT OpenTalk magazine (https:opentalk.iit.it).
The first part of the interview was meant to maximize the participants' positive Comfortability level (e.g., complimenting them), and the second part sought to maximize their negative Comfortability level (e.g., interrupting, ignoring and misunderstanding them). After receiving advice from the journalists involved in this project, actions meant to shape the flow of the interview were included (waiting for the proper time to introduce a new topic to let the interviewee assimilate the situation). Additionally, it was decided to not let the topic itself influence the participants' Comfortability, but to focus on the effect of iCub's behavior. Hence, questions related to sensitive topics were avoided asking them instead about their hobbies, up-to-date news, and professional career. Given the line of questions, only researchers unrelated to HRI were recruited. Figure 1 shows some of the multimedia devices involved, as well as the interviewer's (iCub) and interviewee's (human) position; who were the only ones present in the room. The experiment followed a Wizard of Oz technique (WoZ); i.e., the experimenter controlled the robot from an adjacent room (monitoring the situation through an USB camera and ambient microphone). Additionally, 2 HD and 2 USB cameras (two pointing at the interviewer and the other two at the interviewee), a condenser microphone, and the Shimmer sensor were also included to monitor physical and physiological features. As introduced before, one of the goals of the experiment is to collect data to analyze Comfortability linking it to expressive (i.e., facial and/or corporal expressions) and/or physiological (i.e., hearth rate, galvanic skin response, temperature, etc) signals. Thus, all these devices along its mutual synchronization are needed.

Comfortability Measurement
To assess if iCub is capable of impacting the interviewees' Comfortability, two self-reports were collected: 1) during the interview, iCub asked the interviewees four questions regarding their feelings (see the orange bars of Figure 2); and 2) after the interview, the interviewees were asked to fill a questionnaire to report their Comfortability (following a 7-point Likert scale) regarding specific robotic actions recalled to them in random order (see Figure 2).
Additionally, the data recorded during the interview will be annotated by external observers matching specific reactions to concise Comfortability values. Up to this point, even the data has not been labeled yet, the naturalness and differences among reactions have been confirmed.

Procedure
As we suspect that Comfortability might be related to people's personality and attitude towards robots, participants were asked to complete the TIPI [6] and RoSAS [3] questionnaires some days in advance.
The day of the interview and once they signed the consent form approved by the Comitato Etico Regione Liguria (the assigned Ethical committee), the participant was accompanied to the interview room. As soon as the door was opened, iCub was facing them in "alive mode" (i.e., breathing, blinking and able to follow their face). Then, the participant was accommodated in the chair in front of iCub, fitted with the Shimmer sensor (on their hand), and informed that the interview would be recorded in one shot and they had to remove their mask (security measure regarding the SARS-Cov-2 pandemic) after the experimenter had left.
From another room, the experimenter controlled the robot's behaviour by pressing specific keyboard's keys. After a key was pressed, an action (dialogue plus movement) was executed. The movements were created by specifying joints' positions (iCub owns 53 degrees of freedom) in time with the dialogue, trying to mimic human natural expressions. The whole interview was entirely scripted, thus experimenters controlled only the actions' timing of execution. In addition to the basic actions (executed for all the participants), special actions, included to make the interaction more realistic and fluent conveying the impression of an intelligent robot behavior, could also be triggered. For example, if the participant did not understand iCub's speech, the experimenter was able to repeat that part. To make it more natural, before repeating the same action, the experimenter could include a sentence like "As I was mentioning" or "Again". Also, in case the participant asked any question or their answer was shorter than expected, the robot was capable of intervening (i.e., informing them they were not allow to do so, and asking them "Could you elaborate more?", respectively).
Once the interview finished and the participant filled the Comfortability self-report, the experimenter debriefed the aim of the study justifying iCub's strange behaviour.

PRELIMINARY RESULTS
At the moment, three researchers of different departments of the same institution (plus two internal researchers for the pilot study) have been interviewed (∼20 min each). They were unaware of the experimental goals of their participation, thus they came prepared to disseminate their research achievements. Figure 2 illustrates the Comfortability levels reported by the researchers through the post-questionnaire self-report. Each bar represents a question associated to an interview's key-point (an specific action performed by iCub). Basically, Q1-Q12 comprehend the first part of the interview (meant to trigger positive Comfortability), and Q13-24 the second part (meant to trigger negative Comfortability). It is interesting to notice that even with such a small sample, the interviewees' Comfortability levels vary along the interview and get closer to the expected values.

Self-reports
In fact, Q1 ("How did you feel when iCub said 'Hello, thank you very much for participating! I am very happy that you are here!'") and Q10 ("How did you feel when iCub said 'I am really enjoying talking with you! could I get a selfie with you at the end of the interview?'") were the questions closer to an Intensity = 7 (which means being Extremely Comfortable); and Q18 ("How did you feel when iCub didn't understand your point and made you repeat it four times?") and Q20 ("How did you feel when iCub remained in silence for some seconds, and then said 'Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo'?") were the ones closer to an Intensity = 1 (which means being Extremely Uncomfortable).
Moreover, the bars highlighted in orange show the actions in which iCub asked: "How are you feeling about talking to me?" (Q11), "Are you comfortable?" (Q12), "How are you feeling about this interview?" (Q23), and "Would you like to keep talking with me?" (Q24).
The reported values are in line with the participants' qualitative answers during the interview (e.g., Q11 vs. Q23: "It is fun/a strange experience"; Q12 vs. Q24:"Yeah, sure/why not?"). It seems that when someone is asked directly about their current Comfortability (e.g., Q23/24) rarely a negative comment is reported, despite other reports (e.g., Q13-22) show otherwise. This underlines the importance of using other alternative measures, such as analyzing their reactions (e.g., facial expressions or body movements) through videos.

Visual Reactions
Considering the participants' reactions, it seems that their facial expressions and body movements changed significantly depending on the robot's actions. Looking at Figure 3, two of the multiple reactions associated to the question Q9 (https:// youtu.be/ WMJ_H-pXYoo):"iCub asked about how the world might change as a consequence of the pandemic and then said: 'I've never thought about this perspective. I am sure it could trigger interesting discussions among the magazine viewers'" and Q14 (https:// youtu.be/ 3lnj48rpfqg):"iCub said 'It smells really bad here, is it you?'" are shown. What it would have been expected from these actions, was to elicit positive Comfortability in the first one (1.) and negative Comfortability in the second one (2.). Nonetheless, these results show that 1. was reported with an intensity of 4 (which means being neither Comfortable neither Uncomfortable); and 2. was reported with an intensity equal to 3 (closer to being Extremely Uncomfortable). Additionally, the participants' expressive reactions were found to be natural and relevant to the internal state under study (which will be unraveled in further studies).

DISCUSSION AND FUTURE WORK
This paper presents a real, robot-guided interview as a means to discover the physical and physiological cues related to Comfortability.
It is expected that the actions performed by the humanoid robot iCub will evoke natural and varied Comfortability levels on the interviewees, which was confirmed by the preliminary quantitative results. From the qualitative reports (i.e., answers to iCub's questions during the interview, and verbal feedback expressed to the experimenters) the following aspects were noticed: 1) it seems that the interviewee's educational background is strictly related to the way they might be impacted by the robot's actions. For example, one of the participants possibly because of working activities in the field of AI, did not express reactions similar to the others; 2) the recorded frames revealed that the expressions we believe might be related to extreme negative Comfortability levels arose when the participant was listening to and/or expecting an answer from the robot; and 3) when the interviewees observed iCub's "mean" behavior in the second part of the interview, they occasionally seemed to inhibit their facial movements.
The results obtained in this preliminary experiment are promising even though they were collected on such a small sample. Once enough data is collected, the interviewees' perceived behaviors (i.e., visual, auditory and physiological features) will be study in depth (associating those to certain Comfortability values); and subsequently, an annotated database, which might serve robots and probably other agents to detect human Comfortability, will be built.