Exact image super-resolution for pure translational motion and shift-invariant blur

In this work, a special case of image super-resolution problem where the only type of motion is global translational motion and the blurs are shift-invariant is investigated. The necessary conditions for exact reconstruction of the original image by using finite impulse-response reconstruction filters are developed. If the number of low-resolution images is larger than a threshold and the blur functions meet a certain property, then a reconstruction filter set for exact super-resolution can be generated. If the conditions are not fully satisfied, then the high-resolution image is reconstructed with some error.


I. INTRODUCTION
UPER-RESOLUTION image reconstruction can be defined as the process of constructing a high-quality and high-resolution image from several shifted, degraded, and under-sampled ones. In areas such as medical imaging and satellite imaging, where multiple frames of the same scene can be obtained, super-resolution is proven to be useful. Also, multiple frames in a video sequence can be utilized to improve the resolution for frame-freeze or zooming purposes.
In the literature, super-resolution is treated as an inverse problem, where the high-quality and high-resolution image to be obtained is linked to the under-sampled images by a series of operators such as warping, blur, decimation and additive noise. Some notable super-resolution studies include stochastic reconstruction methods, projection onto convex sets (POCS) approach, iterative back-projection, and others. Excellent tutorials about the subject with emphasis on difficulties and future directions can be found in [1] and [2]. Also a number of special journal issues on super-resolution image reconstruction provide collections of recent work on the topic [3], [4].
In some applications, where the motion is controlled and there is no local movement, the only type of motion within the low-resolution image sequences is translational motion. For example, the scanner resolution can be increased by scanning the document more than once with slightly changed initial points. Also in some video sequences, the scene is static and image sequences are obtained by translational motion of the video camera. There are works in the literature which consider this special super-resolution case [5], [6].
In this work, the conditions for the existence and uniqueness of finite-impulse response restoration filters for exact reconstruction of the high-resolution image in case of pure translational motion (or no motion) and shift-invariant blur are derived. If the number of the low-resolution images is larger than a threshold and the blur functions meet a certain property, then a set of restoration filters can be constructed for exact high-resolution image reconstruction even in the absence of motion.
The paper is organized as follows: In Section 2, the conditions for the existence and uniqueness of perfect restoration filters are derived. In Section 3, given that the conditions are met, a perfect reconstruction method is presented. In Section 4, some simulation results are shown in order to validate the propositions and some conclusions are drawn in Section 5.

II. EXISTENCE AND UNIQUENESS CONDITIONS FOR 2D FIR
RECONSTRUCTION FILTERS Fig. 1 shows the observation model that relates the observed low-resolution images to the original high-resolution image [7]. The desired image, f(n 1 ,n 2 ), and the observed images, g k (n 1 ,n 2 ) (k = 1, …, K), are linked through linear operations such as geometric warp, blur, sub-sampling, and additive noise (not shown in the figure). K is the number of observed low-resolution images. The geometric warp operator, which is denoted by W k , is the representation of the motion that occurs during the image acquisition process. It may consist of global or local translation, rotation, etc. The second linear operation is the blurring process. It results from factors such as relative motion between the imaging system and the scene, out of focus, point-spread function of the sensor, and so on. It is generally modelled as a linear shiftinvariant finite impulse-response (FIR) two dimensional filter, h k (n 1 ,n 2 ). The aliased low-resolution image is then generated by sub-sampling the warped and blurred high-resolution image.
To recover the high-resolution and blur-free image, a set of finite impulse-response reconstruction filters, w k (n 1 ,n 2 ), is applied to the low-resolution images as in Fig. 2. In this section of the paper, we will investigate and determine the conditions that lead to the existence of such restoration filters. The following assumptions and notation are used throughout the paper: and it is merged into the blur function as described in [6]. -The size of each one of the blur kernels is AxA, the size of each restoration filter is BxB, and the downsampling rate is D s for both horizontal and vertical directions. It is straightforward to extend the following discussion to more general cases (rectangular blurs and restoration filters, etc.).
represents the restoration filter coefficients. To determine the existence and uniqueness conditions for the restoration filters, the low-resolution images must be expressed in terms of the original image, the blur operators and the sub-sampling operator, and the estimated image must be expressed in terms of the low-resolution images and the restoration filters, both in vector-matrix notation. Let us define the input vector f(n 1 ,n 2 ) as the lexicographically ordered (A+B-1)x(A+B-1) portion of the original image that is centered at (n 1 ,n 2 ) such as: The input vector is a row vector and its length is (A+B-1) 2 . The data vector, g(n 1 ,n 2 ), can be defined as the concatenation of the K data sub-vectors g k (n 1 ,n 2 ) that are obtained from the degraded images where each sub-vector is formed by lexicographically ordering BxB portion of the corresponding degraded image centered at (n 1 ,n 2 ) such as: where k = 1, …, K. The data vector is a row vector and its length is KB 2 .
The blur matrix H is constructed from the coefficients of the blur functions. It is defined as: where the size of S xxx is D s xD s , the size of S xx is BxB, the size of S x is B 2 xB 2 , and the size of S is KB 2 xKB 2 . The rate of S xx 's that appear on the main diagonal of S x is 1/D s . Note that the sub-sampling matrix described here does not reduce the size of the image, rather it leaves the relevant pixels as they are and makes the others zero.
Based on the definitions above, the input-output relationship of the degradation model described in Fig. 1 in vector-matrix notation is as follows: To reconstruct the original image, a set of FIR filters are applied on the degraded (blurred and sub-sampled) images as shown in Fig. 2. Let us define the restoration filter vector w as the concatenation of K vectors that are formed by the coefficients of the corresponding adaptive filters such that: Combining (1) and (2), the following condition must be satisfied for perfect reconstruction: where e α,β is defined as a column vector of length (A+B-1) 2 whose only non-zero element is determined by the shift (α, β).
For the system in (3) to be consistent, i.e. for the existence of a restoration filter vector that satisfies (3), H·S must be full row-rank. Recall that the size of H·S is (A+B-1) 2 xKB 2 . To determine the conditions that lead to system consistency, first consider the case where D s =1, i.e. no sub-sampling. In this situation, S is the unity matrix of size KB 2 xKB 2 and the problem reduces to the multi-channel deconvolution task. The full row-rank conditions for D s =1 are stated as follows [8]: -The 2-D Z-transforms of h 1 (n 1 ,n 2 ), h 2 (n 1 ,n 2 ), …, h K (n 1 ,n 2 ) must be co-prime. (For the definition and consequences of coprimeness, please refer to [8].) -H must have more columns than rows, i.e. KB 2 ≥ (A+B-1) 2 .
Returning to our case where D s > 1, we can use the inequality about the rank of multiplication of matrices: The inequality in Eq. (4) states that for H·S to be full rowrank, the rank of S must be at least equal to the rank of H, i.e. (A+B-1) 2 . Considering the fact that S is a diagonal matrix whose non-zero entries appear only on the main diagonal as 1's, its rank is equal to the number of 1's that it contains.
where ⎣x⎦ denotes the smallest integer larger than x. As a result, the existence condition for D s > 1 case turns out to be as follows: Some direct consequences of the inequality in (5) can be stated as follows: • The uniqueness of the solution is satisfied when the inequality in (5) becomes equality. • K must be at least D s 2 +1.
• Motionless super-resolution is possible as long as the co-primeness condition and the inequality in (5) are satisfied.

III. AN EXACT SUPER-RESOLUTION ALGORITHM
From (3), the restoration filter vector can be found as follows: where pinv(X) represents the pseudo-inverse of matrix X. Let us denote w that is obtained in (6) as w α,β to indicate that it is determined by the shifted version of the original image. The estimated image can be written as follows: where * denotes the 2-D convolution operator and w k α,β (n 1 ,n 2 ) denotes the restoration filters in rectangular form. The following theorem will be given without proof: , n n f is obtained by (7). Theorem 1 states that by using w α,β , only a portion of the (shifted) original image can be reconstructed. The proof resides in the relationship between the sub-sampling matrix and the low-resolution images. A method to reconstruct the • Given A and K, find B using Eq. (5).
• Construct H and S.

IV. SIMULATION RESULTS
To validate the discussion above, a number of computer simulations are performed on a 512x512 Lena image. The image is first blurred, then sub-sampled. Motion is not considered (note that motion is not necessary as long as the conditions are satisfied). Blurring and sub-sampling of the original image is repeated several times to obtain a number of low-resolution images. The parameters of the blur functions are chosen randomly. Different simulations are performed for different blur kernel sizes, down-sampling factors and different number of low-resolution images. For each case, we tried to reconstruct the original image by using the method described in the previous section. Some results for D s = 3 are given in Table I. The first column of the table is the number of low-resolution images, and the second column is the blur kernel size (assumed to be the same for all blurs). The third column is the size of each restoration filter obtained by (5), and the last column is the mean-square error between the original image and the reconstructed image for the corresponding case. In the table, "n.e." denotes for "non-existent", and "n.a." denotes for "not applicable". It is demonstrated that as long as the number of low-resolution images exceeds the square of the downsampling rate (9 in this case), a set of 2-D FIR restoration filters can be constructed such that exact super-resolution is possible. From the table, it can be deduced that if the number of low-resolution images increases, then the minimum length of restoration filters for exact super-resolution decreases, as expected. Visual results are not necessary to be shown, because when the existence conditions are satisfied, exact reconstruction occurs, when they are not satisfied, no image can be constructed at all. Similar results were obtained for other down-sampling rates, and also when different original images were used.

V. CONCLUSION
In this work, the existence and uniqueness conditions for finite impulse-response restoration filters for exact image super-resolution in the case of pure translational motion and shift-invariant blur are derived. If the conditions are satisfied, a method for exact reconstruction is proposed. Simulation results demonstrate that as long as the conditions are met, exact super-resolution is possible even in the case of nonexistent motion.