CID2013: A Database for Evaluating No-Reference Image Quality Assessment Algorithms

The CID2013 Camera Image Database consists of real images taken by consumer cameras and mobile phones. It is developed to provide useful tool to allow researchers target more commercially relevant distortions when developing processes of objective image quality assessment algorithms.

The CID2013 database consists of 480 evaluated images captured by 79 imaging devices (mobile phones, DSC, DSLR) in six Image Sets. Note that the actual number of images in the database is 474. In Image Set II, Device 6 is evaluated twice as we wanted to test inter-observer reliablity. The scores are later combined into a single MOS value as the two evaluations correlated strongly.

If you use this database in your research, we kindly ask that you follow the copyright notice bellow and cite the following paper:

Virtanen, T., Nuutinen, M., Vaahteranoksa, M., Oittinen, P. and Häkkinen, J. “CID2013: a database for evaluating no-reference image quality assessment algorithms”, IEEE Transactions on Image Processing, vol. 24, no. 1, pp. 390-402, Jan. 2015. [pdf]

Method

The images are evaluated by 188 observers using Dynamic Reference (DR-ACR) method (explained below). A separate scale realignment ACR data consisting evaluations from 34 observers is also included that allows to combine the data from the six image sets

In other respects the DR-ACR method resembles very much a basic Absolute Category Rating (ACR) method (ITU-R 500-11), except the observers saw a slideshow of all the other images in the test depicting the same scene before every evaluation (See DR_demo.mp4). By seeing the other images in the test setup as reference the observers were more aware of the total variation of quality represented within a single image set. This improved their evaluation as they didn’t need to save the far ends of the scale in case there would be even more better or worse image later on the experiment. The DR-ACR method is explained in detail in:

Mikko Nuutinen, Toni Virtanen, Tuomas Leisti, Terhi Mustonen, Jenni Radun, Jukka Häkkinen (2014)  A new method for evaluating the subjective image quality of photographs : dynamic reference Multimedia Tools and Applications 75:  4.  2367-2391 Dec.

Database contains consumer camera images and their subjective evaluations in mean opinion score (MOS), sharpness, graininess, lightness and color saturation scales. It includes the complete raw data and background information from the naïve observers used to evaluate the images. Subjects’ vision was controlled for the near visual acuity, near contrast vision (near F.A.C.T.) and color vision (Farnsworth D15) before the participation. They received movie tickets as a reward. Outlier removal is made for mean opinion score (MOS) evaluations using ITU-R 500-11 recommendations to ease out the implementation of the database.

Material

The images in CID2013 are intended to represent typical photographs that consumers might capture with their cameras. The photographed scenes were based partly on the Photospace approach described by I3A (CPIQ Initiative Phase 1 White Paper: Fundamentals and review of considered test methods, I3A, 2007) The I3A CPIQ project has migrated under IEEE.

The test environment

The room has been covered with medium gray curtains to diffuse the ambient illumination. Fluorescent lights (5800K) were positioned behind the monitors and reflected from the back wall covered with grey curtain to create dim and uniform ambient illumination in the room. The light hitting the monitors measured below 20 lx. The subject’s viewing distance (approximately 80 cm) was controlled by a line hanging from the ceiling, and they were instructed to keep their forehead steady next to the line. Because of the display size, images were scaled to a size of 1600 x 1200 pixels using the bicubic interpolation method. Eizo ColorEdge CG241W, with 1920x1200 pixel resolution, monitors in was calibrated to sRGB having target values of: 80 cd/m2, 6500K and gamma 2.2 using EyeOne Pro calibrator (X-rite co.).

-----------COPYRIGHT NOTICE STARTS WITH THIS LINE------------

Copyright (c) 2014 The University of Helsinki
All rights reserved.

Permission is hereby granted, without written agreement and without license or royalty fees, to use, copy, modify, and distribute this database (the videos, the images, the results and the source files) and its documentation for any purpose, provided that the copyright notice in its entirely appear in all copies of this database, and the original source of this database,Visual Cognition research group (www.helsinki.fi/psychology/groups/visualcognition/index.htm) and the Institute of Behavioral Science (www.helsinki.fi/ibs/index.html) at the University Helsinki (www.helsinki.fi/university/), is acknowledged in any publication that reports research using this database. Individual videos and images may not be used outside the scope of this database (e.g. in marketing purposes) without prior permission.

The database and our paper are to be cited in the bibliography as:

-----------------------------------------------------------------------------
Virtanen, T., Nuutinen, M., Vaahteranoksa, M., Oittinen, P. and Häkkinen, J. “CID2013: a database for evaluating no-reference image quality assessment algorithms”, IEEE Transactions on Image Processing, 2014, In press.
-----------------------------------------------------------------------------

LIMITATION OF LIABILITY

UNIVERSITY OF HELSINKI SHALL IN NO CASE BE LIABLE IN CONTRACT, TORT OR OTHERWISE FOR ANY LOSS OF REVENUE, PROFIT, BUSINESS OR GOODWILL OR ANY DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE COST, DAMAGES OR EXPENSE OF ANY KIND HOWEVER CAUSED OR HOWEVER ARISING UNDER OR IN CONNECTION WITH THE USE OF THIS DATABASE.

THE UNIVERSITY OF HELSINKI SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE DATABASE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF HELSINKI HAS NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.

THIS AGREEMENT SHALL BE CONSTRUED AND INTERPRETED IN ACCORDANCE WITH THE LAWS OF FINLAND, EXCLUDING ITS RULES FOR CHOICE OF LAW.

-----------COPYRIGHT NOTICE ENDS WITH THIS LINE------------