Hypercolorimetric multispectral imaging system for cultural heritage diagnostics: an innovative study for copper painting examination

The aim of this work is to test the application of a new multispectral imaging system, named Hypercolorimetric Multispectral Imaging, on two little 17th century oil paintings on copper in order to support the restoration activities. Hypercolorimetric Multispectral Imaging is a non-invasive, rapid and diagnostic technique that allows in situ accurate and reproducible spectral reflectance measurements between 300nm and 1000nm to obtain seven monochromatic very high spatial resolution images (36 megapixels starting from RAW format). The acquired images are transformed into radiometric and colorimetric measurements, consisting of 7 monochromatic images of spectral reflectance and one colorimetric image. All these calibrated images constitute the base for further processing performed through a dedicated software that implements a number of functions. In the present paper, a subset of those functions has been used. Specifically: Principal Component Analysis, spectral clustering, spectral mapping, multiband contrast enhancement and edge detection. Combining calibrated images of different spectral regions acquisitions, it was possible to extract relevant information about the state of conservation of the two copper paintings and further significant details were readable compared with the data coming from each single acquisition. The Hypercolorimetric Multispectral Imaging acquisition process revealed to be fast allowing to be performed during the cleaning stage of the paintings. The imaging nature of the analysis allowed to compare and map different areas of the surfaces producing degradation maps of the painting layers, which represents a precious decision-making tool for conservators.


Introduction
The practice of painting on metal surfaces to produce art objects has started in ancient times and its popularity has increasingly grown in Europe, especially in the Netherlands and in Italy, during the 15th and 16th centuries [1,2]. Probably, the development of rolling machines and techniques as etching and engraving during the 15th century, as well as the growth of enamelling on copper, contributed to the usage and availability of metals for artistic purposes. Painting on copper surfaces has attracted artists for a variety of reasons of both practical and aesthetic nature: they are expected to be more durable than paintings on canvas or wood, the support being rigid (thus, not being susceptible to tearing or cracking) and being made from metal, which neither suffers from biological attack nor does it exhibit a strong response to fluctuations of temperature and relative humidity, but only from minor dimensional changes due to temperature in normal indoor conditions. Such features account for the surprisingly good state of preservation that some paintings still exhibit today. In addition, the rigidity of the support and their usually small format greatly contributed to their portability [3].
Copper supports are non-absorbent [4]; they can preserve rich and highly saturated colours, even if applied in very thin layers/glazes, due to the oil binder that is not absorbed to a great extent. This, in combination with the smoothness of the metal substrate, which provides a reflective support, results in paintings with a very unique, translucent and luminous look often described as "jewel-like" [5].
A wide range of factors influence durability, adhesion, uniformity and protective performance of the copper substrate as a support for oil paint: metal purity, alloying, crystallographic structure and surface inhomogeneity bear on the initiation and speed of copper corrosion processes. These characteristics are themselves affected by the production and manufacturing methods, due to residual stresses left by deformation [6].
In order to investigate and restore such complex and unique artefacts, efficient, portable and cost effective noninvasive methods have been required for diagnostics. Such techniques can extend the amount of information obtained from the artwork analyses with traditional methods and limit the number of invasive testing required [7][8][9].
In recent years, the multispectral imaging (MSI) technique has undergone significant development by virtue of an increasing amount of camera technologies that are rapidly finding applications in different fields, including pharmaceuticals, agriculture and food quality control, as well as cultural heritage diagnostics and, in particular, for paintings conservation [10][11][12][13][14][15][16][17]. The main advantages of the MSI technique are material identification and mapping of artistic materials, to reveal retouching and restored areas in the painting, to differentiate degradation patterns and collect data with the purpose of authentication [18][19][20]. It is also recognized that the full diagnostic potential of MSI may be improved by implementation of robust data processing algorithms for digital images, supporting various aspects of conservation, allowing to observe materials below the surface and, through the combination of different spectral range imaging, to gain latent information on painting technique and ancient restorations, as well as for evaluating artwork authenticity.
One important aspect of spectral imaging is that it generates a large amount of data, which may be difficult to interpret; in this perspective, post-processing methods are needed in order to fully exploit the information contained in the image cube. The main achievement of MSI data processing is to distinguish and identify materials from their spectral signatures and to spatially group pixels with similar characteristics. A large variety of methods are available to perform this task, primarily based on spectral transform and multivariate statistic techniques such as Principal Component Analysis (PCA), spatial domain filtering and neural networks tools [21]. PCA has been widely used for art conservation applications, either as a stand-alone technique or to reduce the dimensionality of datasets using linear algebra techniques prior to another classification algorithm [18,22,23], whereby multiple variables are analyzed to evaluate the contribution of each variable to an observed result. Clustering techniques can classify data based on a comparison of the spectral character of each point in the image with the spectral character of every other point; this permits patterns or clusters to emerge from within the data, indicating areas of compositional similarity and returning a spatial distribution of components [19].
To the authors' knowledge, this work presents the first ever application to copper painting analysis of the Hypercolorimetric Multispectral Imaging (HMI) developed by Profilocolore Srl. The proposed system was chosen because it has very high radiometric (better than 95%) and colorimetric precision (better than ΔE = 2). This precision, combined with high spatial resolution, provided a useful support to the restoration activities of copper paintings [24].
With respect to well-established multispectral methods, HMI allows to effectively combine the multispectral channels, due to the calibrated data obtained after the acquisition through a modified camera. The possibility of combining the multispectral channels, after calibration, supplies great advantages in terms of information that can be derived from the acquired data.
Another great advantage of HMI, particularly relevant in artworks examination, is the rapidity of acquisition (only three shots) without the necessity of power supply [24].

Artworks' description and cleaning approach
The objects of this study are two 17th century oil paintings on copper plate, one representing a Virgin of the Immaculate Conception ("painting A", 14.5 × 19.5 cm) and the other a Praying Virgin with Sleeping Child ("painting B", 15.3 × 20 cm) (figs. 1-2). The two small copper plates were originally positioned on the external surface of the two shutters pertaining to a wooden chest drawer from the S. Giuliano monastery in L'Aquila (Italy). Both copper plates were altered from previous restoration actions, specifically surface treatments, and also affected by mechanical deformations of the thin and well-polished metallic support. Furthermore, they both showed significant chromatic alteration of the upper finishing layer (with a probable original finishing still present), consisting in the browning of the shellac coating also used for the maintenance of the wooden shutters and carbonaceous deposits due to the prolonged indoor use of candles.
A further relevant conservation issue was the presence of detached areas and losses in the painting layer, which is made up of an overlapping of several thin oil glazes; the presence of an underlying oil preparation layer seems to  be probable, also according to specific literature. Horovitz [1] and Stock [25] both mentioned a specific practice for the copper plate preparation, which is to apply a layer of linseed oil. They noted that this could be done in order to clean the surface from any residues of substances used to roughen the plate, being a practice borrowed from etchers; or as Stock [25] noted, for practical purposes, as its presence is thought to facilitate the subsequent application of the preparation layers. Stock also linked, though with reservations, the presence of this oil layer to the etchers' practice of applying a special ground, known as a "hard ground", consisting of linseed oil and colophony, prior to etching.
In copper painting restoration great attention is required in the cleaning operation; due to the non-absorbance of the metallic surface, the chosen cleaning agent is not able to homogeneously spread through the support, but it tends to linger on the surface or to be stagnant in the interface between copper and the painting layers, persisting its action on oil binders with an increasing risk of softening and losing the pictorial material. In addition, it is recommended not to use cleaning aqueous systems in order to avoid starting corrosive reactions [26].
For these reasons, some preliminary tests were performed, as usual in restoration works, through the application of organic solvent mixtures [27]. These tests were not effective in removing the surface altered layer. So, a further test was tried using an aqueous system containing a weak chelating agent (1% of tri-ammonium citrate) alternated with methyl-ethyl ketone. In order to avoid the diffusion of the aqueous solution into the painting layers and the copper support, a hydrophobic compound was applied before cleaning. In particular, cyclometicone D5 was used as a silicone-based gel able to prevent the interaction of the aqueous solution with artwork layers. This new approach seemed to effectively remove the altered finishing layer by means of a weak chelating agent.
Velvesil Plus, a commercial gel made of reticulated siloxane polymers, was also used for cleaning the surface areas where the thickness of the altered layer appeared higher and more difficult to be removed with the previously described solution.
The tested systems seemed to be effective in making a highly controlled cleaning of the surfaces avoiding a vertical diffusion of the aqueous solution through the pictorial layer that could affect the painting layers and above all the copper support.

HMI technique
In order to support the evaluation of a proper restoration intervention, it is important to understand the distinctive executive features of a copper art object and the related degradation processes, especially in the cleaning actions that have to be inert and compatible with the metallic support and preserve the original painting layer. For this reason non-invasive diagnostic investigations were led during the cleaning operation to remove the darkened shellac varnish by applying an innovative multispectral system, Profilocolore Hypercolorimeric Multispectral Imaging (HMI) [24,[28][29][30], a portable diagnostic technology for cultural heritage analysis.
The HMI technique is based on the use of an optimized set of imaging sensor, optical filters, radiometric references, natural light or xenon flashes, calibration software and image analysis software. The Profilocolore HMI products are SpectraPick for the image acquisition and calibration, and PickViewer for the image analysis.

SpectraPick -Image acquisition and calibration system
The imaging sensor is based on high-end Nikon 36 megapixel D810FR camera, modified under a Nikon-Profilocolore project, to reach the full spectral range of sensitivity of silicon sensors from about 300 nm to 1000 nm [24,30].
The camera is equipped with interchangeable lenses to adapt to distance and dimension of subject, and with 2 multiband pass filters with spectral transmittances that are quite complimentary and, combined with the Bayer filters of the modified camera sensor, are able to correctly sample the whole 300-1000 nm range.
Lighting plays a key role and needs to cover, with a continuous spectrum, the 300-1000 nm range. Among different possible solutions, xenon flashes have been chosen. In particular, Nikon SB910 flashes were used with front plastic lenses removed to allow also the UV wavelength to be emitted. The UV-induced fluorescence was obtained filtering the flashes light with a UV band pass filter with a cut at 380 nm, and UV-IR cut filter (400-700 nm) in front of the camera.
The whole multispectral and colorimetric acquisition is extremely fast and requires only three shots. The radiometric references consist in a number of white patches (to sample any uneven lighting) surrounding the subject and of a 36 patches colour-checker. These references are built using colour's samples out of the NCS -Natural Colour System c catalog and their spectral reflectances are accurately measured in Profilocolore lab in the range of 220 to 1050 nm at 0.7 nm accuracy (Instrument System Spectroradiometer CAS 140 CT and dark room) [24].
The calibration procedure follows three steps: 1) white patches analysis to achieve correct white balance and even exposure across all the subject; 2) running an Artificial Intelligence based optimizer that finds the mathematical function to translate camera digital values of the colour-checker into radiometric and colorimetric measurements; 3) applying the calibration to the acquired images and producing 7 monochromatic images, 16 bit TIFF format, containing the spectral reflectance at 350, 450, 550, 650, 750, 850 and 950 nm, and 1 AdobeRGB TIFF 16 bit colour image. The achieved precision across the whole 36 megapixel image is better than 95% on the spectral reflectance images and colour error less than CIE2000 ΔE = 2 for the colour image ( fig. 3).
If the acquisition process includes also the UV induced fluorescence, or any other imaging data, the calibration software is able to align these further images to the previous ones, with the precision of one pixel. All the calibration and alignment process requires only few minutes and can be performed in field to immediately check results against the subject.

PickViewer -Multispectral image processor
The HMI acquisition and the calibration system provide the user with spectral reflectance and colour coordinates for each one of the 36 megapixel of the captured scene. Those data are calibrated and are absolute, depending only on the spectral characteristics of the surface. The nature of the data allows performing many kinds of analysis to investigate the subject through Profilocolore developed PickViewer , a software tool with custom and specific image processing function able to handle multispectral images.
The PickViewer features are: -integration of any other imaging data (fluorescence, X-ray, thermal, etc.); -multichannel images viewer, pan, zoom; -any pixel colorimetry and spectral reflectance read-out; -mapping by colour, spectra, arbitrary channels; -principal components analysis; -neural network based clustering; -colour and spectral signature database; -two ways mapping by database entry; -any channel to RGB false colours visualization; -channels math, indexes and normalised contrast; -calibration and colour-checker test.

Results and discussion
The two copper paintings were investigated with the HMI technique during the cleaning of the altered finishing varnish which entirely covered the original painting layer. The whole acquisition process, including the UV-induced fluorescence image, required only a few minutes and, after calibration, it provided spectral reflectance and colorimetry.  The potentiality of the technique can be seen in fig. 5, where visible image during cleaning and the result of image processing (spectral similarity and edge detection) are shown, immediately highlighting some losses of pictorial materials. In the present section the most relevant results are reported and discussed in terms of relevance for the conservative aspects of the restoration work. The criterion for choosing the combination of different images was related to the information that conservators expected from the elaboration. The conservation status and the cleaning result were the main interesting aspects for the restoration work: for this reason, UV reflectance and fluorescence images were selected for processing.
In painting A, contrast enhancement operations performed on the combined UV reflectance and the inducedfluorescence images ( fig. 6) have highlighted all the punctual discontinuities in the painted layer, which are mostly concentrated along the edges of the small copper board, where the original frame was before restoration procedures began.
Other contrast enhancement algorithms were applied on the second principal component from PCA performed on the UV-induced fluorescence image combined with the colorimetric blue component of the VIS image (normalized difference: . This elaboration allowed to clearly distinguish areas where the altered varnish was completely removed, from areas still covered by residuals of the varnish (fig. 7). The still uncleaned areas are well visible on the upper part of the Virgin veil and on some letters in the left upper corner of the painting, appearing lighter in respect to the rest of the digital elaborated image (fig. 7). The comparison of spectral profiles between the two areas (uncleaned and cleaned) highlights the differences (see fig. 8).
Because these uncleaned areas were both not valuable in the VIS image, the conservators were able to completely remove the altered varnish.   In the HMI investigation of painting B, principal components from PCA run on the induced-fluorescence image were processed with the IR2 image (i.e., the band centered at 850 nm) and the radiometric blue component of the VIS image in order to highlight detachment spots and small mechanical hollows in the metallic foil ( fig. 9). These are particularly visible in the left part of the painting background (which, at the moment of multispectral acquisition, had been cleaned from the overlaid altered varnish) and in correspondence of the foil's borders originally covered by the ancient frame ( fig. 9).
A further contrast enhancement operation, processing the first three components of PCA performed on the UVinduced fluorescence and the RGB image in the UV band, permitted to better investigate the right part of the copper painting, highlighting alteration (probably in the ground layer or in the metallic support) on the top right part of the Virgin's veil ( fig. 10).
Thanks to the possibility of having highly reliable spectral reflectance for each pixel of the image, after identifying a little area where the painting layer is missing, pixels of spectral similarity within a given threshold were identified on the whole painting ( fig. 11). Edge detection output on the resulting map has been extracted and overimposed to the original VIS image, allowing to highlight all the areas where the original pictorial material was missing, including the ones not clearly discernible to the naked eye, because of the adjoining painting spots presenting the same colour ( fig. 12).
Considering the high resolution and radiometric precision (see fig. 3) of the HMI imaging technique and the small dimensions of the copper painting, these acquisitions and damage mappings are extremely precious for studying the state of conservation of the painting during the restoration intervention, allowing to clearly distinguish original and restoration materials, alteration products and deformations in the metal support.
These results helped to complete a noteworthy restoration project and to preserve an accurate calibrated and high-resolution digital documentation for future conservation and monitoring actions [31,32].

Conclusions
Hypercolorimetric multispectral imaging systems provide a powerful tool for non-invasive identification and characterization of pigments, substrates and conservative treatments of cultural heritage objects, allowing non-destructive analyses for paintings restoration and for conservation and documentation purposes. The present paper shows the great potential of the HMI system as a non-invasive, rapid and reproducible method to investigate art materials and, above all, to support the conservation process, especially to address a reasonable number of micro-invasive sampling points for confirming material characterization with traditional analytical techniques.
Residuals of the altered shellac finishing layer, as highlighted in the ultraviolet fluorescence images and spectral profiles (see fig. 8), were removed by restorers, thanks to the HMI analysis that detected them by means of spectral and spatial HMI resolution and allowed to address and complete the cleaning process.
The HMI system can be considered as a valuable support to evaluate the state of conservation of painting through the identification and mapping of damages and discontinuities in the painting layers and copper support in order to support restorers and monitoring process before, during and after restoration.