Signal Processing: Image Communication

High dynamic range (HDR) images require tone reproduction to match the range of values to the capabilities of a display. For computational reasons and given the absence of fully calibrated imagery, rudimentary color reproduction is often added as a post-processing step rather than integrated into tone reproduction algorithms. In the general case, this currently requires manual parameter tuning, and can be automated only for some global tone reproduction operators by inferring parameters from the tone curve. We present a novel and fully automatic saturation correction technique, suitable for any tone reproduction operator (including inverse tone reproduction), which exhibits fewer distortions in hue and luminance reproduction than the current state-of-the-art. We validated its comparative effectiveness through subjective experiments and objective metrics. Our experiments confirm that saturation correction significantly contributes toward the perceptually plausible color reproduction of tonemapped content and would, therefore, be useful in any color-critical application.


Introduction
Recent advances in both capture and display technologies allow images of a much wider dynamic range to be photographed, manipulated, and displayed; better capturing the light of natural scenes and giving artists unparalleled freedom. Although HDR standards and workflows are being defined and have begun to be adopted, they are not 5 yet mainstream. As such, HDR technologies currently coexist with more prevalent consumer imaging pipelines [4]. HDR data often needs to be compressed for display on most current displays, a process known as tonemapping or tone reproduction. In contrast, existing low dynamic range (LDR) data may need to be expanded or reconstructed in order to fit the capabilities of emerging HDR display devices, a process 10 known as inverse/reverse tonemapping (ITM) [5]. In both cases, the aim is to preserve the appearance and information content of an image as much as possible while ensuring that it can be displayed on the chosen display device. To achieve that, tonemapping and inverse tonemapping algorithms typically operate on the luminance of the image with little to no consideration for the color information present, leading to noticeable 15 changes in the color appearance of the image, as shown in Figure 1.
Commonly, luminance-compressed images acquire an over-saturated appearance when only the luminance channel is processed [6,7]. Image appearance models, which can be seen as tone reproduction operators with integrated color appearance management [2], are designed to reproduce color appearance, but they require calibrated images, 20 precise knowledge of the scene in which the image was taken as well as measurements of the viewing environment and the display device itself. This makes these algorithms very useful in color-critical applications, but their requirement for measurements coupled with high computational complexity due to spatially varying processing limits their general applicability. 25 Some solutions exist for correcting saturation mismatches after tonemapping [7]. This leads to a computationally efficient correction, although hue and luminance shifts may be introduced. Moreover, they require manual parameter selection which is strongly image and tone reproduction operator dependent. Recently, a subjective study was conducted for defining an automatic model to derive the parameters necessary for such 30 corrections, but only allows parameters to be predicted when the tone compression or expansion function is global [6]. In this paper, we therefore, present an efficient and effective color post-processing technique with the aim to relieve the user from having to set parameters, while being applicable to any form of image processing, whether spatially varying or not. This has the additional benefit that our post-processing can 35 be applied even if the input image was manually touched-up, including but not limited to manual dodging and burning. Our work offers the following contributions and advantages: (1) Our novel algorithm is based on recent advances in perceptually linear color-space and saturation computation. (2) Irrespective of the applied image processing technique or tonemapping operator, our algorithm is fully automatic and able to 40 recover an accurate reconstruction of image saturation. (3) We take the gamut boundary of the output color space into consideration, leading to lower hue shifts and significantly lower luminance distortion. (4) We evaluate our algorithm by means of a subjective experiment and objective metrics, revealing that our algorithm reproduces saturation significantly better, as well as significantly reduces luminance distortions 45 than the current state-of-the-art.

Hue and Saturation Correction
Tonemapping aims to compress the dynamic range of images and prepare them for display. Typically, this happens through a non-linear transformation of the luminance are the one specified in their original work [8] and [1]. Note that each operator changes the relationship between input and output luminance in different ways leading to different types of saturation and hue shifts.
channel. The aim of tonemapping is then two-fold; images need to be processed so that 50 their absolute luminance range is compressed, but pixel relations also need to be altered to maximize visible detail, therefore changing the contrast in the image. Changes to contrast and luminance, however, often lead to changes in the appearance of colors in the image and specifically in their saturation and hue. Furthermore, different tonemapping algorithms alter luminance and contrast in vastly different ways (Figure 2), pre-55 venting a simple correction parameter to work for all cases. Thus, our algorithm is designed to correct the image's appearance while minimizing luminance and contrast modifications without requiring the user to set any parameter [9].

Algorithm Overview
The input to the algorithm consists of two images given in a linear RGB color space:  active non-linear management of saturation values to account for the Hunt effect [10].
Although HDR images are given in linear units, since in most cases accurate radiometric data is not available, their luminance values are inherently inaccurate. As such, we focus on contrast changes between the two input images and therefore normalize 70 both M t and M o before converting them to the IP T color space, which has better hue uniformity than CIE L * a * b * and HSV color spaces [11]. Recently, a variant of the IP T space for HDR images, known as hdr − IP T space, has been proposed. In this new space, the power function in IP T has been replaced with the Michaelis-Menten function to improve the behavior of the color space for very low and very high lumi-75 nance levels [10]. We decided to not use this color space to avoid using different color spaces for tonemapped and HDR images.
As we need separate access to lightness, hue and colorfulness, we then convert to a cylindrical color space akin to CIE L * C * h * . This space is based on IP T and therefore we refer to it as the ICh space, where I encodes lightness, C represents colorfulness 80 and h is a measure of hue. The lightness channel I is not further processed because this was the main purpose of the preceding tonemapping operator. The hue in the tonemapped image h t is subsequently set to the hue h o of the original image, restoring any hue distortions that may have arisen due to gamut clipping during tonemapping.
The quantity that needs to be matched between the HDR and tonemapped images is 85 saturation (s). However, the aforementioned cylindrical color space produces colorfulness (C). Saturation is defined as colorfulness relative to lightness, i.e. s = C/I.
However, a recent proposal to define saturation as colorfulness relative to the full magnitude of the stimulus, i.e. s = C/ √ C 2 + I 2 [12], provides more accurate results for our application.

90
After the saturation is adjusted on a per-pixel basis, our adjustment is modulated to avoid creating out-of-gamut pixels. Finally, the corrected image is converted back to the RGB color space and gamma corrected as the final step. The workflow of our algorithm is illustrated in Figure 3 and discussed in detail in the following sections.

95
Accurate color processing often benefits from the use of a perceptually linear and decorrelated color space. An obvious choice in this case would be the CIE L * a * b * , as it aims to be perceptually uniform, meaning that the color difference between two pairs of colors with the same Euclidean distance between them in the CIE L * a * b * space will be perceived equally different. Although this is a useful property, images modified 100 in the CIE L * a * b * may acquire hue shifts [11].
To address this particular shortcoming, an alternative color space known as IP T was proposed, where I encodes lightness information while the P and T channels (standing for protan and tritan responses) encode red-green and yellow-blue opponent dimensions respectively [11]. Like CIE L * a * b * , it is defined as a transform from XY Z 105 tristimulus values. The perceptual uniformity offered by CIE L * a * b * is preserved in IP T , but additionally changes in colorfulness do not induce hue shifts (see Figure 4), making it better suited for our purposes. The IP T space forms the foundation for our algorithm. However, we wish to manipulate perceptual correlates of hue and saturation, and for this reason we convert from IP T space to a cylindrical version of IP T , 110 which we term ICh, as explained next.

Appearance Correlates
To convert from IP T to a cylindrical color space ICh, we follow the standard procedure and leave the I channel unchanged while setting hue h and colorfulness C as described in [9]: h = tan −1 (P/T ) (1) Saturation s is commonly computed as s(C, I) = C/I. Recently, however, an alternative formula was proposed that follows human perception more closely [12]: Note, however, that to our knowledge application of this formula in ICh is novel; its development was centered around CIE L * C * h * . The merit of using this formulation is assessed in Section 3.1.

Saturation Correction
Since saturation for a given pixel depends both mathematically and perceptually on its lightness, it can be expected that after tonemapping an image, its saturation values will be changed. Most tonemapping operators modify only luminance and leave chromatic information unchanged, which inevitably alters the relation between lightness and saturation in the image, therefore leading to an over-saturated appearance. Tonemapping typically maps luminance values in a non-linear manner. As a result, although the absolute luminance levels of the tonemapped image are likely to be lower than the original HDR scene if displayed on a conventional monitor, the relative luminance of many pixels will be increased compared to their surrounding pixels. According to the Hunt effect, these pixels will then appear even more saturated, requiring additional correction. Noting that we have already normalized both tonemapped and HDR images before converting to ICh, to account for the Hunt effect, we begin by scaling colorfulness according to the relative lightness of the original HDR and tonemapped images: Then, using (3), we compute the ratio r between the saturation of the original and tonemapped image, albeit that we compute the latter using C t , i.e. after accounting for the Hunt effect: This ratio is then applied to colorfulness C t as a second factor to find the colorfulness appropriate for the tonemapped image: For convenience, in the following, we will refer to the full adjustment factor as: Dependent on the used tone curve, the colorfulness of some pixels may increase while for others it may decrease. This may even happen in the same scene, so that light a. RGB gamut boundaries pixels may gain in colorfulness, while dark pixels may lose colorfulness. This is a desired effect, but it does mean that light pixels may be moved toward and over the 120 gamut boundary as a result of applying a scale factor that is larger than 1. To prevent this, it is possible to readjust the value of r prior to applying it to the colorfulness of the tonemapped image. This is discussed in the following section. Finally, we reset the hue by copying values from the HDR image (h c = h o ). Together with the corrected colorfulness C c , it is combined with the lightness channel of the tonemapped image 125 I c = I t to produce the final corrected result, which can then be converted back to RGB and then gamma corrected for display purposes.
It should be noted that in this work we assume that both the input and the output target encoding -both in terms of color space and non-linearity-is BT.709 [14]. If a different output encoding is necessary, e.g. BT.2020 [15], which may be the case 130 when inverse tonemapping content towards HDR, we propose that the transformation towards the desired encoding is performed after our processing, using the linear output of our algorithm.

Gamut Correction
When pixels in the tonemapped image M t are near or on the gamut boundary, increases 135 in colorfulness may lead to undesirable hue and luminance changes in the resulting image due to gamut clipping in the conversion from ICh to RGB. Several gamut mapping techniques have been proposed, to solve this problem in the past [16]. These techniques have been mainly developed to overcome the limited mismatch between color gamuts, i.e., LDR image and display. Only recently techniques dealing with 140 HDR content are starting to appear [17]. However, the aim of our gamut correction is two folds. The first is to perform this operation on the LDR content. The latter is to have a computationally efficient solution that does not add unacceptable extra overhead on the color correction algorithm, while maintaining acceptable quality.
To reduce the occurrence of such clipping, we developed a further adjustment to the colorfulness correction r . The conversion between RGB and ICh is non-linear, and as such there is no easy way to determine what the maximum colorfulness is given a specific lightness level I and hue h. This can be seen in Figure 5 where we have plotted the vertices of an RGB cube before and after transforming to ICh. Although there may be analytic or sample-based solutions to describe the corresponding ICh gamut boundary, or distance metrics in complex volumes could be devised, these are not computationally efficient and would add a disproportionate cost to the main algorithm. Therefore, we propose a simplified and approximate algorithm to determine how far a given color is removed from the gamut boundary. Figure 5 shows that the RGB gamut is by definition cubic, and in our case it is located within the unit volume. Therefore, we compute the shortest distance of each pixel in the tonemapped image to the gamut boundary in RGB space: where the factor of 2 normalizes the distance d. The approximation we make is that 145 we assume the distance to the gamut boundary in RGB space to correlate with the distance to the gamut boundary in ICh space. We, therefore, use distance d to directly adjust the colorfulness scale factor.
We achieve out-of-gamut detection by converting a copy of the tonemapped image after the hue reset back to RGB and compute d based on that. Wherever hue reset has created gamut related problems, we will have a value of d less than 0, meaning that at least one of the color components have gone out of the [0, 1] range. We now have a choice as to whether we would accept a hue shift or sub-optimal saturation for these out-of-gamut pixels. We could reduce colorfulness until these pixels become representable in the output RGB gamut, thereby minimizing hue shifts. On the other hand, we could accept these hue shifts and keep our saturation processing as accurate as possible. Either approach would be viable. However, to demonstrate the utility of our algorithm, we have chosen for the latter by simply clamping negative values of d to 0. On the basis of d, we can now adjust our correction factor if it were to move pixels too close to the gamut boundary, or beyond. Rather than hard clipping, it is often desirable to gently reduce the processing for pixels near the gamut boundary. We have found that a straightforward rational function allows us to effect such a gentle roll-off: The steepness of the roll-off is controlled by the constant 0.01, a parameter that was determined empirically. Note that this choice of this parameter is not critical, so that changes to this parameter value would not unduly affect the results. The contrast adjustment then becomes: This function effectively produces a non-linear interpolation between the desired adjustment factor and a factor of 1 when near the gamut boundary.

150
As mentioned previously, in this work, we consider that the input content is encoded in the BT.709 gamut, and in this gamut correction step, we aim to preserve it. If the target output gamut is larger, e.g. BT.2020, to compute the distance in Equation (8), we first need to transform the image M t from BT.709 to the larger color space, passing by XYZ, and according to the transformations described in the respective standards. After

Evaluation
To assess the performance of our algorithm, we compressed the dynamic range of many challenging scenes with different tonemapping operators. We then processed 160 the results with our color correction method and compared our results against both the automatic and manual versions of Schlick's and Mantiuk's algorithms [6,7]. Since we assume that the color correction method is applied as a post-process and therefore has no direct access to the tonemapping algorithm, for Schlick and Mantiuk's techniques we estimate the tone curve from the image pair directly so that their parameters can be 165 directly estimated.
In the following, we begin by showing in Section 3.1 the merit of several design decisions taken during the development of our algorithm. In Section 3.2 we show sideby-side comparisons with existing techniques as well as usage scenarios. Then, in Section 3.3 we assess and compare lightness and hue reproduction, which can be mea-170 sured objectively. This is followed by an evaluation for the case of ITM in Section 3.4.
Finally, the comparative performance of saturation reproduction is assessed with a psychophysical experiment, which is discussed in Section 3.5.

Evaluation of Design Decisions
In this paper, we argue that the choice of color space is critical to the success of our 175 method. An obvious choice for color processing would be CIE L * a * b * . However, consistent with the literature [11], we found that it has relatively poor behavior, particularly in blue regions. This can be seen in Figure 6 where we have applied our color processing in both CIE L * C * h * (left), derived from CIE L * a * b * , and ICh (right).
Here, we have also applied two different formulations for saturation, the commonly 180 used s = C/I and the more recently proposed (Equation 3). We see that in CIE L * a * b * with the standard saturation computation the sky takes on a purple tinge, an effect we have seen in other images as well. This is fixed by using the new saturation formula, but here the final result is too saturated. Using ICh space no undue color shifts are present in Figure 6, but we observe that here the standard saturation formula 185 ICh CIE L*C*h* C/I leaves the image somewhat too desaturated. These effects are seen to a greater or lesser extent in many images. We have therefore chosen to do our processing in ICh space, using the more recent saturation computation, as explained in Section 2.

Results and Comparisons
The algorithm was implemented in MATLAB, running on an Apple Macbook Pro with  can be corrected. Figure 7 demonstrates this with an example where the image is selectively desaturated using a mask. Although most of the image is overly saturated, a region in the shape of the logo is almost achromatic. The mask in this case has only af-200 fected saturation and not the luminance or contrast. Our approach corrects the colors in the image such that the desaturated logo becomes almost invisible after our correction.
Note that if both the HDR and the tonemapped images are individually normalized, the tone reproduction process does not universally reduce the image's contrast. Instead, some pixels are reduced in level, whereas others are increased. As a result, some pixels 205 require a commensurate decrease in saturation, while others need their saturation to be increased. Figure 8 shows that the effect of our method is that materials can be correctly reproduced, irrespective of tone reproduction operator. The gold leaf on the  Figure 8: The Memorial image was tonemapped using six different tone reproduction operators. The saturation was then corrected using our method, as well as Schlick's and Mantiuk's algorithms with parameter automation enabled [6]. Our method desaturates darker areas in the image more, following color perception, while lighter areas are preserved or even enhanced. As a consequence, the gold plating on the wall (inset) maintains a gold appearance in our results. wall still appears as gold for instance; an effect that is difficult to reproduce with other methods that tend to create more washed-out colors.  [1] than with the photographic operator [8].

215
Ideally, saturation correction aimed at tone reproduction should alter the saturation or colorfulness of the image without affecting the luminance or hues. Specifically, the hues of the original input image should be preserved throughout the process, while the luminance and contrast information should be defined by the tonemapping process applied. To assess whether our algorithm achieves that, we have evaluated our results Typically, color differences are computed by simultaneously considering both luminance and chromatic information. In our case, a metric capable of separating luminance, saturation and hue is necessary as we are only interested in preserving two of these appearance correlates. Further, we compare hue information relative to the in-225 put HDR image, while luminance information is evaluated relative to the tone mapped image.
Although, commonly, color difference measurements are performed in the CIE L * a * b * color space, it is known that CIE L * a * b * is not hue-linear across all hues [11], making this space not ideal as a basis for measuring hue differences. The IP T color space addresses some of the limitations of CIE L * a * b * , but to further optimize the space for computing color differences, an adjusted I P T color space was developed by Shen et al. [18]. This space is scaled and rotated with respect to IP T such that color differences are directly comparable with other color difference metrics, while preserving hue linearity. It is computed from IP T coordinates as follows: In I P T , a cylindrical space is then computed as discussed in Section 2, where lightness ∆I and hue ∆h differences can be computed. As hue is defined on a circle, we compute ∆h for a given pair of hues h t and h c as follows: Figure 10 shows aggregated results over a dataset of 99 HDR images, drawn from the HDR photographic survey [3], which were tonemapped twice; once with Li et al.'s algorithm [1] (right) and once with the photographic tone reproduction operator [8] 230 (left). Lightness differences were evaluated against the tonemapped results in each case, while hue differences were evaluated against the input HDR image, to ensure that hue was preserved throughout the process. The GC and NGC abbreviations represent the gamut correction (Section 2.5) and no gamut correction cases, respectively.
We observe that lightness is significantly better reproduced in our method; slightly Images were tonemapped with the photographic operator [8] and with Li et al.'s algorithm [1]. Error bars indicate a 95% confidence interval. The GC and NGC abbreviations represent the gamut correction (Section 2.5) and no gamut correction cases, respectively.

Inverse Tonemapping
Our analysis so far has focused on the tone compression case. Nevertheless, the grow-245 ing availability of commercial HDR displays has increased the need for color-managed solutions for expanding the dynamic range of existing LDR content. Recently, Bist et al. [20] study has shown that for gamma-expansion based ITM operators the ideal saturation parameter setting s is 1.25 when using the Mantiuk's luminance preserving formula. Our method is equally suitable for ITM applications [21]. We present a 250 more holistic solution that irrespective of the ITM operator is fully automatic and able to recover an accurate reconstruction of image saturation. At the same time, we take the gamut boundary of the output color space into consideration, leading to lower hue shifts and significantly lower luminance distortion. ITM operators tend to create undersaturated results for the same reason that tonemapping operators tend to saturate too 255 much. Figures 11 and 12 show that our algorithm is able to restore saturation in this case. Note that for visualization purposes each result was subsequently tonemapped with the photographic operator [8] .
In addition to visual comparisons, we evaluated the performance of the different cor-  al. [24]. Aggregated results are given in Figure 13 and example visualizations of dif-265 ference maps are shown in Figure 14. The second row shows the lightness differences, while the last row shows the hue differences.

Psychophysical Evaluation
While luminance and hue performance can be assessed with suitably chosen color difference metrics, the aim of saturation correction algorithms is to alter saturation taking set of psychophysical experiments.

Experimental design
To assess the saturation performance, we designed a 2-alternative forced-choice ex- [7], our proposed approach and [6] for 99 images. Our method was tested both with and without the gamut clipping step. Error bars indicate a 95% confidence interval.

Hue Difference
Lightness difference Figure 14: Example results with different correction methods for ITM as well as corresponding lightness (second row) and hue differences (last row). ITM results for this example were obtained using the method of Akyuz et al. [22], and are shown after tonemapping for visualization.
can emit up to 4000 cd/m 2 . To allow prolonged stable and calibrated use, we used a 280 peak luminance of no more than 2500 cd/m 2 . The background of the stimuli was set to 18 cd/m 2 while the peak luminance for the tonemapped images was 100 cd/m 2 .
The left and right 7 cm of the display were left unused as we have found luminance reproduction to be less accurate in those regions. The display was driven by an Ap- shown in Figure 17. These HDR images were tonemapped with the global version of

Pilot Studies
In the first pilot study, our algorithm with and without gamut correction is evaluated to determine which of these two versions performed best in terms of saturation reproduction. We found that both methods being selected virtually the same number of times.

Main experiment
The task for the main experiment is to match the impression of saturation between tonemapped color processed images and their HDR originals. Therefore, it is not a     they are greater than a critical value R which is computed as in [9]. Figure 15 shows the overall results of our experiment. When we assessed the overall performance, for each tonemapping operator, over all images, we found statistical significance for Li et al.'s 335 operator at significance level α = 0.001. The critical value is R = 53, given u = 144 for 18 participants × 8 images. In this case our method was selected significantly more often. This is visualized in Figure 15 where  Figure 15, is therefore highly statistically significant (α = 0.001). In essence, this means that our algorithm matches the impression of saturation between tonemapped, 350 color corrected images and their HDR originals measurably better than the current state-of-the-art.

Conclusions
Tonemapping tends to be carried out on a luminance channel while leaving chromaticities unaffected. As the appearance of saturation depends on relative luminance levels, 355 ideally saturation should co-vary with luminance when applying tonemapping operators. Nonetheless, it is possible to post-correct saturation mismatches given the input and the output images of a tonemapping algorithm. Based on recent insights into the design of perceptually linear color spaces as well as a recent formulation of saturation, our algorithm provides an effective solution for color post-processing of tonemapping 360 operators as well as manually processed images. Our algorithm is shown to better preserve both lightness and hue information relative to the majority of different tonemapping and inverse tonemapping operators. As our solution is agnostic to the operator used, it can correct saturation after both local and global tonemapping, which is not currently possible with methods relying on estimating the slope of the tone curve. Al-365 though we do not explicitly address video content in this paper, our method can be further extended to handle video content in a temporally coherent manner, assuming that the tonemapping approach is temporally stable. This will be part of future work.