GPU-BASED BACKGROUND ILLUMINATION CORRECTION FOR BLUE SCREEN MATTING

Separation of foreground objects from an almost constant backing color for video applications is still a common problem ([1],[2]). For non-realtime situations there is a wide variety of different powerful mathematical approaches that can deal with most of the matting problems. For SD/HD studio realtime keyers most solutions are not applicable due to their algorithm complexity or high effort in user interaction. Excellent hardware keyers, such as Ultimatte TM work on most occasions, but even under controlled lighting in a blue-/greenscreen matting problems may occur, or creativity is limited by necessary lighting conditions. As a preprocessing algorithm for traditional chroma keying systems, we present a simple background illumination correction based approach for improving matting problems with uneven or poor lit blue-/greenscreens. Using the computational power of GPU computing (GPGPU [3]) the presented algorithm is realtime capable and offers an improvement for achievable mattes quality.


INTRODUCTION
For about sixty-five years the blue screen technique has been the method of choice for background replacement [4]. Using only color information, the blue or green background can be replaced by a different one. In television production this procedure is often referred to as chroma keying which differs in some slight ways from the process used for movie production. The technique is nowadays well-understood and used for a huge number of applications, for instance news or weather presentation, moderation of television journals or virtual studio production. Although excellent realtime keying solutions are available, there are some occasions where they do not provide satisfying results. The lighting of the background plays an important role how mattes will turn out satisfactory. Poorly lit blue-/greenscreens degrade the quality of the matte that can be extracted. Uneven lighting is a common problem, because it is very difficult to get the lighting both bright and uniform across a broad surface, figure 1 demonstrates this problem. Another frequent problem is a too bright or too dark lit background combined with uneven lighting (figure 8). Pulling a matte without high cleanup level is nearly impossible. Close up shots are not so vulnerable to lighting problems, but wide screen shots with a high amount of background and floor are problematic. Resulting mattes without cleanup or special techniques (e.g. automatic rotoscoping from postproduction plugins) show a really poor result. Sample results a shown in figure 3. Refining these raw mattes in a live keying system would lead to really hard edged results, due to a high cleanup level. Other limitations occur in the quality of the remaining shadows. The goal of a realtime capable preprocessing algorithm is to improve the basic image which then can be passed to traditional chroma key techniques. As a possible solution a simple approach using illumination estimation based algorithm is presented. Using graphics hardware for general purpose computations is getting more and more popular, a vibrant community of developers has emerged around GPGPU ( [3]). For demonstration of the computational power of modern GPU architectures the presented algorithm is implemented as shaderprograms (introduction to GPGPU programming concepts [5]).

Illumination estimation
Illumination estimation or shading correction is a very sophisticated subject. Especially edge preserving methods like anisotropic diffusion are common and widely researched. For the presented problem we outline an even simpler method of shading correction, avoiding the solution of complex equation systems, based on single-scale-retinex methods ( [6]). The algorithm is based on the assumption that uneven illumination is an additive low frequency signal ( [7]). Therefore gaussian low pass filtering could be used to extract the illumination distribution across the image. The Gaussian function G(x, y) is defined by (σ defines the effective spread of the function): Convolving an image with a gaussian kernel would delimit the spatial frequencies, resulting in loss of edge definition and averaging of intensity values. For large kernels the resulting image is the information of the present lighting. Normally shading correction is used for the whole image, we present a different approach. First the foreground is coarsely keyed, the hole is filled by a simple inpainting technique, now the gaussian convolution is applied.

General-Purpose Computation Using Graphics Hardware
GPUs are a compelling solution for applications that require high arithmetic rates and data bandwidths. GPUs achieve this high performance through data parallelism, which requires a programming model distinct from the traditional CPU sequential programming model ( [5]). Additionally the gap between the computational power of the CPU compared to the GPU outperforms Moore's law: The used paradigm is called stream processing. The stream programming model exposes the parallelism and communication patterns inherent in the application by structuring data into streams and expressing computation as kernels that operate on streams. Kernel functionality is implemented in high-level-language shader programs, CPU arrays are transferred to the GPU's memory and are represented as textures. Bottleneck is still the download/readback of the data. For outperforming the CPU computational power the data transfer times have to be considered. With the latest chipsets and graphic cards (NVIDIA nForce 680i chipset, GeForce 8800GTX) satisfying transfer rates for SD/HD720p images can be achieved (

ALGORITHM
The presented preprocessing algorithm consists of several buildings blocks, nevertheless the method is clear and efficient.

Figure 5: algorithm flowchart
The necessary steps for creating a corrected image with the computation using the GPU is outlined in the following enumeration: 1. download image to the GPU 2. subsample image 3. pull matte with simple chroma key with high cleanup 4. dilate the matte 5. automatic inpainting 6. gaussian convolution 7. upsample image 8. composite illumination corrected image 9. readback image or pull matte with a shaderprogram After downloading the image to GPU's memory represented as texture, the first step makes use of the assumption that the illumination change is low-frequency. Therefore a subsampling of the image is possible, the low frequencies still remain. Dependent on the subsampling factor (subsampling by a factor of max. 4 is proposed) not only matte pulling and dilatation computation time benefit, furthermore smaller gauss kernels can be used. After this operations foreground elements are determined. Foreground mask and image are now used for a simple inpainting algorithm (overview at [9]). The simplest method for inpainting would be a linear interpolation between the edge pixels of the matte. A more sophisticated fast digital inpainting can be done by repeatedly convolving the region to be filled with a diffusion kernel (e.g. [10]). Convolving an image with a Gaussian kernel is therefore equivalent to isotropic diffusion (linear heat equation). Possible diffusion kernels are (variables (a,b,c) [10]): Figure 7(a) shows the result of the inpainting algorithm. As presented in subsection 2.1 the inpainted image can be used for illumination estimation by convolving with a large gaussian kernel. The optimal size of the gaussian filter is dependent upon the scale of the objects in the image and the overall size. Experiments have shown that as a starting value 1/20 of the image width is recommended. Figure 7(b) shows the result of the illumination estimation which can be used for compositing a corrected image.
The following equation in pseudo code notation present a possible compositing method for blue screen background, where r, g, b are the image color channels, illum is the illumi- Noticeable is the change in the color distribution in rgbspace. Foreground and background clusters separate as expected even better after the illumination correction, the deviation of the background pixel distribution is lower. With a further pass on the GPU easily brightness/contrast or gamma modifications could be done in realtime. The corrected image can now be used for further keying processes while the original image is used for the final composite.

RESULTS AND CONCLUSION
The algorithm presented in this abstract implements an realtime capable and easy preprocessing method for improving the keying result for images with uneven or poor lighting in the background. As shown in figure 10 the mattes with   A perpixel keying algorithm is easy implementable on the GPU with execution times in the range of 1 − 2ms. A Further speed improvement can be achieved by combining several building blocks into a single shader-program and thereby increase the arithmetic operations per transferred word (computational/arithmetic intensity). As a result of an illumination correction the color of foreground objects is changed too and are therefore useless for the final compositing of foreground and background images. For a general keying system a special procedure is proposed (see figure 11. The corrected foreground object is only used for receiving an appropriate matte. For the final composite a preprocessed foreground image with additional color correction is applied. This step yields to more flexibility for adjusting brightness, contrast and color of the individual images. With the illumination correction as preprocessing there is more headroom in the creativity for lighting situations in live productions and therefore the"'poor-lit-bluescreenproblem"' gets a little less problematic.