TWO!EARS Deliverable D4.2 - Specification of feedback loops and implementation progress (WP4: Active listening, feedback loops & integration of cross-modal information; FP7-ICT-2013-C TWO!EARS FET-Open Project 618075)

The workplan of WP4 starts with the following statements.
‘Active listening entails bottom-up data processing as well as top-down mechanisms. To capture them, a framework must be set up that can host suitable feedback loops. It is the general task of WP4 to design an appropriate system architecture for this requirement, investigate meaningful feedback paths, implement them, and finally evaluate them regarding their functionalities. Input from other modalities than the auditory one will also be considered as a source of feedback information, particularly, position, direction and speed of head-&- torso movements (proprioceptive and sensorimotor input), and of identified optical objects (visual input).’

In this context, the consortium has now made final decisions with regard to which possible feedback loops to investigate. Consequently, the process of implementation and evaluation of some of the selected loops has started. These activities belong to Task 4.2, the description of which reads as follows.

Task 4.2 Implementation of feedback loops

‘This is not an isolated task. It can only be performed in close interrelationship with WPs 1–3 & 5 (robotics component). Feedback loops will be designed and implemented following the strictly functional approach outlined on pp. 14/15 of the proposal and based on the in- and outputs of feedback loops as identified in Task 4.1. Implementation of feedback loops will be carried out stepwise, depending on state of development of affected system modules at various relevant stages of system. Stability problems will be investigated. During the implementation phase, the architecture of the Two!Ears system or
at least of some of its modules will have to be suitably modified. Clearly, this process will last during most of project duration. Functional implications of the proposed feedback loops will be traced by analyzing responses to auditory and multi-modal stimuli similar to those used in the final evaluation in Task 4.4. Results will allow for grading feedback loops according to their functional relationships with possible brain mechanisms and thus provide means for their calibration – which would hardly be possible otherwise.’

It is obvious that the implementation of feedback loops depends very much on functional modules that have been set up and delivered by other workpackages. Consequently, the major effort of WP4 will be taken in the last project year. Nevertheless, relevant work is already going on, and some interesting results have already been achieved. Where necessary
modules are not yet available for a fully fletched implementation on the Two!Ears core system, their function and output is simulated. To this end, a virtual testbed, the Bochum Virtual Test Environment (BEFT), is in the process of being developed. A lean version of it, (LVTE), which can already perform a number of relevant tasks, is described in Sec. 2.1 and was uploaded to the Git-repository. The full BEFT will be functional in due time. Nevertheless, it has to be stated that the results achieved so far with regard to auditory feedback are still fragmentary. Actually, this does not come by surprise, since a full integration of the feedback algorithms into the Two!Ears core system becomes only feasible once all necessary modules are at hand. This will be the case during the next months.

In the further course of this document, the sequence of Sections is structured according to a recent list of feedback loops to be considered and explored in Two!Ears.