Sensing passive thermal objects in outdoor scenes for autonomous robots

A physics-based adaptive Bayesian pattern classification model that uses a thermal infrared imaging system to automatically sense passive thermal objects in unstructured outdoor environments for mobile robots is described. The usefulness of thermal infrared imaging as a mobile robot sensing modality is explored, and a set of thermal-physical features used to characterize passive thermal objects in outdoor environments is presented. Objects that extend laterally beyond the thermal camera's field of view, such as brick walls, hedges, picket fences, and wood walls as well as compact objects that are laterally within the thermal camera's field of view, such as metal poles and tree trunks, are considered. Classification of passive thermal objects is a subtle process since they are not a source for their own emission of thermal energy. A detailed analysis is included of the acquisition and preprocessing of thermal images, as well as the generation and selection of thermal-physical features from these objects within thermal images. Classification performance using these features is discussed, as a precursor to the design of the physics-based model to automatically classify these objects.


I. INTRODUCTION
The goal of our research is to enhance an autonomous robot's situational awareness by providing it with the ability to classify smaller objects and structures that cannot be assessed in enough detail by GPS and onboard terrain mapping systems. Our particular interest is in the situation where the bot has already detected an object or structure but now needs to classify it. Thermal infrared imaging gives the ability to identify targets during conditions of limited visibility and it eliminates the need for a light source which could disclose the bot's location. The technology necessary for thermal imaging has just recently become sufficiently portable and inexpensive to mount on small robotic platforms, and because the coming generations of infrared imagers will be yet smaller and cheaper it is important to explore the capabilities of this modality. Classifying passive thermal objects in outdoor environments using a thermal infrared imaging system is a complex problem due to the variation of radiance emitted from the objects as a result of the diurnal cycle of solar energy. Passive thermal objects are not a source for their own emission of thermal energy, and so exclude people, animals, vehicles, etc. Our choices of such natural objects include trees and bushes but not the squirrel running out from behind the tree in Figure 1. Constructed objects include brick walls, wood walls, fences, and metal poles. The ability of passive objects to display a thermal signature depends on the thermal energy received from heat generating sources in the environment, especially the sun, but it also depends strongly on the object's physical composition and surface properties, which we refer to as thermal-physical properties. We intend to design a classification model that retains as much of the original physical interpretation of the information in the signal data as possible throughout the classification process. This will result in a framework that allows the human analyst to understand the reason for a bot's classification of an unknown object by associating the final classification decision with information about the inherent thermal and/or physical properties of the material and/or surface and/or shape of the object found in the original signal data. It also allows for an understanding of those conditions under which thermal imagery is not well suited to distinguish passive objects. Additionally, our approach will ultimately afford bots with the intelligence to automatically interpret the information in signal data to make decisions without the need for an interpretation by humans. The research presented here is a part of work to automate the fusion and interpretation of data streams from various active and passive sensor systems to enable autonomous mobile robot operations in a wide variety of unstructured outdoor environments. We feel that it is the fusion of active and passive sensors that has the potential for the greatest advancements because of the complementary nature of the modalities. Two mobile robots are currently being used to collect systematic ultrasonic and infrared imagery data streams about The College of William & Mary campus, the adjacent restored Colonial Williamsburg, nearby York County, Virginia, and elsewhere. We have used these data sets to explore the behavior of features generated from the signal data of classes of outdoor objects as a step towards designing multi-sensor classification algorithms that afford mobile robots the ability characterize outdoor objects. The research presented here is an extension to our previous work involving sonar sensor interpretation by mobile robots [1]. This research focused initially on the design of algorithms to distinguish common outdoor objects such as trees, poles, fences, walls, and hedges based on features generated from backscattered sonar echoes. Thermal infrared imaging is a complementary technique to classify the same types of objects since both ultrasound and infrared are appropriate for use both day and night. Our research is aimed towards designing a framework that fuses information from the bot's thermal infrared imaging and ultrasonic sensors to perform intelligent actions, such as decision-making and learning. The various test objects and typical thermal images of each are shown in Figures 2 and 3.  surface temperature of low specific heat objects, such as the leaves on hedges, tend to track the availability of solar energy [2]. On the other hand, objects with a high specific heat, such as a tree trunk [3], will tend to heat up and cool down more slowly. Furthermore, for outside objects, windy conditions may enhance convective heat transfer. We have yet to explore systematically the effects of adverse weather conditions (such as rain, snow and ice) but they will affect both heat conduction in objects and their surface thermal properties. Environmental conditions will sometimes exist where the surfaces of a target and surrounding objects will emit approximately the same level of thermal radiance. This phenomenon, known as thermal crossover [2], results in minimal thermal contrast between the surfaces of objects and the surrounding environment. Our mobile thermal imaging system is built around a Raytheon ControlIR 2000B long-wave infrared thermal imaging video camera with key specifications: 320 X 240 pixel resolution, 30 Hz frame rate, 18" x 13.5" field of view (with 50 mm lens) and ferroelectric staring focal plane array detector type. A Samsung Tablet PC with an Intel Celeron 900 MHz processor, 512 MB of RAM, and Microsoft Windows XP Tablet PC Edition operating system allows the operator to capture thermal imagery for sorties up to 2.5 hours in length using a Futaba remote control module to navigate the mobile platform. The analog video signal is transmitted from the camera through a harness cable assembly to a Voyetra Turtle Beach Video Advantage USB video capture device that is attached to the Samsung Tablet PC and converts the composite analog signal from the camera to an 8-bit digital video signal. Thermal imagery was captured on a predefined set of passive outdoor objects during a nine-month period, at various times throughout the days and at various illumination/temperature conditions. The ambient temperature was recorded during each session; weather conditions and recent history were downloaded from a local weather station. The image data was divided into two categories: extended objects and compact objects. The extended objects consist of objects that extend laterally beyond the camera's field of view. Our extended objects ( Figure 2) consist of brick walls, hedges, wood picket fences, and wood walls. The compact objects ( Figure 3) consist of objects that are laterally within the camera's field of view, especially metal poles and tree trunks. During each session, thermal images were captured of each object from two different viewing angles: normal incidence and 45 degrees from normal. The training data consisted of 1060 images acquired during 55 sorties. A second distinct 168-image set of the same objects was subsequently captured at the same viewing angles during nine sorties, and a third data set consisted of the same classes captured at the same viewing angles as the training data, but were not the same objects and not all in the same local area. That blind data set of 132 images acquired during 14 sorties was used to further validate the classifier. Because of both the inherent (and intentional) imprecision of the measurement geometry and the differing meteorological conditions, the second set of imagery acquired on the original objects provides validation testing of the algorithms. The third distinct data set includes objects that humans easily categorize, but different trees, poles, fences, walls, etc. were used to ensure that the algorithms being developed aren't somehow only applicable to the particular objects that were initially selected for this study. The most important step in our pattern classification model design process is thermal feature generation, and since we're dealing with passive objects it's not obvious a priori what features in the infrared images are going to be most important. Such features are typically abstract representations of a passive thermal object that are derived from the given object's thermal image. Our goal is to generate thermal features that not only assist in distinguishing one object class from another but also have a physical interpretation. We consider three types of features: meteorological, micro, and macro, described in Table 1. Micro features are defined so that they include only the object, whereas macro features include some of the background. For example, micro features from a picket fence are calculated for a region which only contains a picket while macro features use a window that includes several pickets as well as the space between them. This allows us to include the physical properties of the picket fence material and then also information about the sizes and spacing of the pickets in the fence.

II. CLASSIFIER DESIGN AND PERFORMANCE ASSESSMENT
The nature of these feature values was explored as a prerequisite to feature selection and performance assessment. A reccurring observation in the two-and three-dimensional feature spaces for both the extended and compact objects is that the data for each object class tends to diverge from the other object classes beginning at a common origin caused by thermal crossover. We see a separation in the object classes that is dependent on the variation in thermal features due to the diurnal cycle of solar energy. Consequently, the origins represent thermal conditions in the environment that are just Amb. temp rate of change (T1) T 1 =

Macro features
First-order statistics right for the feature values to not display much distinction between object classes. Since the shape of each object class's cluster resembles a conical structure, we refer to them as hyperconoidal clusters. In typical classification applications involving controlled environments, the feature values for each object class tend to form compact hyperspherical or hyperellipsoidal clusters with no common origin among the object classes. These applications normally use traditional metrics to choose a set of features for the classification model such as the inter/intra class distance, where the most favorable set of features is the one that results in a large distance between object class clusters (interclass) and small distance between feature vectors within each object class (intraclass). Since our application involves a dynamic outdoor environment, we are dealing with a somewhat more complex situation that will require other methods to identify the most favorable feature vectors. Principal component analysis (PCA) is a traditional feature extraction method for dimensionality reduction of a feature space. As a dimensionality reduction technique, PCA is applied globally over the patterns of all the object classes in the feature space. For a data set of size m consisting of ndimensional feature vectors from all object classes, there exists an eigenvector that not only determines the direction of the maximal variance of the data in feature space but also best fits the patterns in a least squared sense. We perform local PCA on each object class and compute the first principal eigenvector that provides a best fit through the respective object class's hyperconoidal cluster.
Based on our preliminary analysis to assess the quality of the data and eliminate redundant features, we chose a subset of the extended and compact object thermal features to be used in our feature selection process. The goal was to select a set of feature vectors that result in the lowest error rates when combined with either the Bayesian, K-Nearest-Neighbor (KNN), or Parzen classifier. We computed the error rates for each classifier combined with every combination of features across all possible dimensions (i.e., exhaustive search feature selection method) because we had access to the DoD high performance computing facilities and thus had the luxury of a brute force approach to make sure that we weren't coloring the results with some initial assumptions about the expected behavior. For each classification model (i.e., classifier plus feature vector), an average error rate was computed using the resubstitution, holdout, and leave-one-out error estimation methods. The resulting average error rates were compared to determine the feature vectors that present the lowest error rates for each type of classification model across the extended and compact object classes, respectively. There appears to be no "single optimal" feature vector but a "set of most favorable" feature vectors associated with various classifiers for both the extended and compact object classes. Moreover, it appears that the set of most favorable feature vectors may contain contributions from all the feature types: meteorological, micro, and macro. Consequently, our chosen thermal-physical features display potential to uniquely represent the extended and compact objects and will be used in our continued research to design a physics-based model to automatically classify passive thermal objects [4].
We used these most favorable feature vectors to design and implement a novel model, shown in Figure 4, that outperforms the traditional KNN and Parzen classifiers for our specific application. The design of the adaptive Bayesian classification model is based on the observation that the thermal patterns for each class of passive thermal objects display a unique behavior about an eigenvector that projects through their respective hyperconoidal cluster. The behavior is characterized by the normal distances between the patterns and eigenvector for each object class. Various distance functions are derived based on these normal distances. These distance functions are integrated into the likelihood function of the Bayesian classifiers to form an adaptive Bayesian classifier. We have found that the combination of specific sets of adaptive Bayesian classifiers and most favorable feature vectors yield exceptional classi-fication performance for a given object class. Each set of adaptive Bayesian classifier models acts as an expert in making classification decisions on patterns from their respective object class. It turns out that we can form a committee of experts for classifying patterns from a specific object class. Consequently, one committee of experts may perform exceptionally on specific unknown patterns where another classifier is deficient, and vice versa. By combining each committee of experts into one classification model, we are able to exploit the expertise of each committee and complement the overall performance of the classification model. We further increased the confidence level in our model's classification decisions by integrating a dynamical windowing technique that lets each committee of experts decide on class assignment by considering information collected from multiple window sizes of the thermal image of an object. Additionally, we incorporated rules into our model that must be satisfied before the bot is authorized to make a classification decision. If all the rules are satisfied, the bot is authorized to assign a class to the unknown object within its field of view and proceed with the next required action in the intelligence algorithm. On the other hand, if a rule is not satisfied, the bot must reject a class assignment and capture another thermal image of the unknown object for classification, perhaps from another viewing angle.

III. CONCLUSION
We have designed and implemented a physics-based adaptive Bayesian pattern classification model that uses a passive thermal infrared imaging system to automatically characterize passive thermal objects in unstructured outdoor environments for mobile robots. The resulting model outperforms the traditional KNN and Parzen classifiers [4]. The framework of our classification model could also be used in other applications requiring the characterization of unknown objects based on features that witness variations due to natural cyclic events. For instance, our model could be integrated into classification applications that use RGB video to generate features from the visible images of objects in outdoor scenes that depend on illumination from the sun. The adaptive Bayesian classification model could also be used during quality control inspections on assembly lines or perhaps in a variety of thermographic Nondestructive Evaluation (NDE) applications.