Learning and executing multiple tasks together vs. one task at a time

The analysis of a complex scene requires the application of a considerable number of visual tasks, well beyond what is performed in current models. Multiple objects in the scene need to be recognized and located, together with their properties and inter-object relations. A single scene may contain a large number of objects, and objects parts, together with their properties and relations, and the total number of object classes, properties and relations for a human-level scheme is estimated to be in the tens of thousands1. It is consequently infeasible to learn all the tasks simultaneously or to extract the full structure of complex scenes. As discussed below, problems of combinatorial generalization further prohibit the simultaneous learning and execution of multiple tasks simultaneously. We conclude that it is often desirable to learn one task at a time. On the other hand, during scene analysis, to obtain a fast response it is desirable to perform multiple tasks simultaneously whenever possible. We propose an approach in which the model can be instructed to learn and to execute specific visual tasks, either individually or in combinations. We show howit is possible in this model to learn tasks individually, but then perform multiple tasks together without any additional learning. The ability to perform multiple tasks within a single step is subject to limitations, which are demonstrated and analyzed in our work.