%%%%% This folder contains data pertaining to the paper "Integrative processing in artificial and biological vision predicts the perceived beauty of natural images" by Sanjeev Nara and Daniel Kaiser The following explains the folder contents. For all further information, please contact Daniel Kaiser: danielkaiser.net@gmail.com %%%%% The folder contains all relevant data, stimuli, results, and analysis code (in Matlab). The contents of the folder are explained below. Note that all folders are provided as .zip files. They need to be unzipped into the same top-level folder to run the analyses. %%%%% Behavioral rating data The folder /behavior/ contains the behavioral data for the four studies: - For studies 1-3, res.beauty contains the beauty ratings for the 250 images in an image x participant matrix - For study 3, res.complexity and res.order contains the complexity and order ratings - For study 4, the mean beauty ratings from Brielmann & Pelli (2019) are provided. Individual ratings can be found here: https://github.com/aenneb/OASIS-beauty %%%%% Stimuli The folder /stimuli_places1/ contains the 250 images used in Experiments 1&2 The folder /stimuli_places2/ contains the 250 images used in Experiment 3 The folder /stimuli_oasis/ is empty and needs to be filled with the OASIS images, which can be obtained here: https://www.benedekkurdi.com/#oasis The folders with the same name and the appendix "_shine" contain images matched for luminance (lum_) or luminance and contrast (hist_) Note 1: the OASIS image numbers need to be left-padded with zeros before running the analysis scripts! Note 2: To create the "_shine" images for OASIS, you need to download the OASIS images and pass them to the SHINE toolbox (see the script /supplementary_analysis/R1_create_shine_images.m) %%%%% DNN models The folder /vgg16_places365/ contains the vgg16 trained on Places365 used in the main analyses The folder /vgg16_imagenet/ contains the vgg16 trained on ImageNet used for supplementary analyses (Note that the usage of these models in Matlab requires the Deep Learning Toolbox!) %%%%% DNN results The folder /results/ contains the results of the DNN analyses, separately for the Places365 and ImageNet DNNs. For the Places365 DNN used for the main analyses, the results files ('*res*') contain the following information: - cnn.corr: Correlation between the activation patterns for the whole image and the average of the halves (integration; with flipped sign) - cnn.sim: Correlation between the activation patterns for one half and the other half (part-based similarity) - cnn.l2: Activation strength for the whole image (L2 norm). Note that the spatial scale here is redundant, as only the whole image is used. All three fields are organized in the following way: - cnn.corr{scale}, with spatial scales from 1-5, from fine to coarse - Each entry in the cell array is then an image-by-layers matrix (Note that Places1 refers to the stimulus set used in Studies 1&2, and Places 2 refers to the stimulus set used in Study 3!) The folder additionally contains the correlation between the DNN measures and the behavioral data ('*prediction'). The main results file (using Places365) is cnn_prediction.mat. The file contains the following information: dat.c: correlations between DNN measures and beauty ratings (dat.p proviedes the corresponding p-values, uncorrected) - The field is a cell array with four entries, reslecting the four studies - Each cell array contains another cell array with five entries, reflecting five different analysis, in the following order: 1) correlations between integration and beauty 2) correlations between part-based similarity and beauty 3) partial correlation between integration and beauty, with part-based similarity partialed out 4) correlations between whole-image L2 norm and beauty 5) partial correlation between integration and beauty, with whole-image L2 norm partialed out - Each cell in turn contains a cell array with 5 entries for the spatial scales, from fine to coarse - This cell finally contains a vector, where each entry reflects a network layer. - Hence: dat.c{study}{analysis}{scale}(layer), the same notation is used in the following description, with the ordering of conditions as explained above dat.data contains the data used for computing the correlations, in the format: - dat.data{study}{scale}{layer} contains an image-by-variable matrix, with the following variable ordering: 1) beauty rating 2) integration measure 3) part-based similarity measure 4) L2 norm for the whole image dat.corr_co contains the correlations of beauty, integration, and the complexity and order ratings (Study 3), in the format: - dat.corr_co is a variable-by-variable-by-layers matrix, with the following variable ordering: 1) beauty rating 2) integration measure 3) complexity rating 4) order rating dat.c_co contains the partial correlations of integration and beauty ratings, with complexity or order partialed out (dat.p_co proviedes the corresponding p-values, uncorrected) - dat.c_co{1}{1}{scale}(layer) is the correlation with complexity partialed out - dat.c_co{2}{1}{scale}(layer) is the correlation with order partialed out dat.r_crossval contains the correlations between predicted and real data in the crossvalidated linear model analysis (dat.p_crossval proviedes the corresponding p-values, uncorrected) - dat.r_crossval{study}{1}(scale) %%%%% DNN results for image manipulations The folder /results/ also contains the results of the DNN analyses where input images where manipulated in targeted ways. Analogously to the previous analysis, the *manipulations_res* files contain the values of the DNN-based measures for each of the image manipulations, with the same conventions as above. The file cnn_prediction_manipulations.mat file contains the predictions of beauty ratings from the integration measure for the manipulated images. Specifically: - dat.c{study}{1}{scale}(manipulation, layer): the correlations with beauty ratings - dat.p{study}{1}{scale}(manipulation, layer): the corresponding p-values - dat.r_crossval{study}{1}(manipulation,scale): correlations between predicted and real beauty ratings in the linear modeling analysis - dat.p_crossval{study}{1}(manipulation,scale): the corresponding p-values - dat.pred{study}{scale}(manipulation,:): the predicted beauty ratings in the linear modeling analysis - dat.stats_across_manipulations{scale}(manipulation,manipulation,study): Pairwise tests (p-values, corrected) between correlations of real and predicted ratings across all manipulations - dat.stats_against_original{scale}(manipulation,study): Tests (p-values, corrected) between the correlations of real and predicted ratings obtained for the original images and the manipulated images %%%%% fMRI results Aggregated fMRI data are found in the folder /fMRI_analysis/, in the file fmri_integration.mat, which contains the following variables: - roi_names: cell array with the names of the 7 regions in the order they appear in the results - fmri: integration measure, with flipped sign, for the fmri, in a participant x region x image matrix - fmri_whole: realibility measure of the whole image, in a participant x region x image matrix - fmri_univariate: univariate activation for the whole image, in a participant x region x image matrix - fmri_parts: part-based similarity measure, in a participant x region x image matrix The folder additionally contains image_names.mat, a cell array with the names of the images used in the fMRI experiment, to match them to the Images in /stimuli_places2/ %%%%% fMRI raw data Raw fmri data are found in the folders /raw_fmridata_s*/. Each folder contains one participant, with the following contents: - functional scans: 10 x 144 *.nii files for the scans required in the 10 experimental runs (all volumes are realigned and corregistered to the participant's structural T1 scan) - region_masks: binary masks for each region of interest: V1, V2, V3, V4, OPA, MPA, PPA (separate masks for each hemisphere) - condition files: multiple condition files in SPM format, containing the onsets for each scene (onsets), the condition indices (names) and the event durations (all 0) (Note: the condition indices in the multiple condition files are: 1-32: full scenes, in order of the files names, 33:64 & 65:96: scenes halves in the same order) %%%%% fMRI results fMRI results are found in the folder /fMRI_analysis/, in the file fmri_prediction.mat, organized in the following way: - dat.img_ind: Indexes of the images used in the fMRI experiment, to match them to the Images in /stimuli_places2/ - dat.corr{1}(region): correlations between the fmri integration measure and beauty ratings (dat.p provides the corresponding p-values, uncorrected) - dat.corr_corrected{1}(region): correlations between the fmri integration measure and beauty ratings, with whole-image reliability partialed out (dat.p_corrected provides the corresponding p-values, uncorrected) - dat.corr{2}(region), dat.corr_corrected{2}(region): Same as above, for the part-whole similarity measure - dat.corr{3}(region), dat.corr_corrected{3}(region): Same as above, for the univariate activation measure - dat.data{region}: provides the data used for computing the correlations, in an image X variable matrix, with the following variable ordering: 1) beauty ratings 2) fmri integration 3) fmri whole image reliability 4) fmri part-whole similarity 5) fmri univariate activation - dat.fmri_cnn_corr: The is a 2xlayer matrix for the correlations between integration in the PPA and in the DNN, with the following 2 columns: 1) correlation 2) p-value %%%%% Supplementary Results Note 1: For details on the Supplementary analysis, the reader is referred to the analysis scripts in /Supplementary_Analysis/ Note 2: For further details, please get email Daniel Kaiser. Category-analysis based on k-means clustering: - cnn_places365_classification.mat: Outputs from the last FC layer for dividing the image sets into categories - cnn_prediction_categories.mat: Results for the category-based analysis Symmetry analysis: - image_symmetry.mat: horizontal / vertical symmetry values for all images - cnn_prediction_symmetry.mat: Results for the symmatry analysis Random splits on the 4x4 and 8x8 grid: - cnn_places365_randomsplits4x4_res_*: DNN measures for the randomsplits in a the 4x4 grid - cnn_places365_randomsplits8x8_res_*: DNN measures for the randomsplits in a the 8x8 grid - cnn_prediction_randomsplits4x4.mat: Results for the randomsplits in a the 4x4 grid - cnn_prediction_randomsplits8x8.mat: Results for the randomsplits in a the 4x4 grid Variance partitioning analysis: - cnn_prediction_vpa.mat: Results of the variance partitioning Predictions based on a linear model trained on all DNN features: - cnn_places365_regression_res.mat: Results for predicted versus real beauty ratings %%%%% Main analysis scripts Note 1: All analysis scripts can be run in Matlab from the folder they are placed in. Note the the Deep Learning toolbox is required and that the OASIS images need to be added, see above. Additionally, some helper functions may be needed (see bottom of this document.) Note 2: Scripts with an "R1_" prefix were added during the revision #1 of the paper - run_dnn_vgg16_places365.m: computes the integration measure, part-based similarity measure, and whole-image L2 norm for all studies - run_dnn_vgg16_imagenet.m: same as above for the ImageNet DNN - analyze_dnn_places365.m: relates the DNN measures to the behavioral ratings - analyze_dnn_imagenet.m: same as above for the ImageNet DNN - R1_create_shine_images.m: create luminance and luminance&contrast matched images using the SHINE toolbox (requires the toolbox: https://link.springer.com/article/10.3758/BRM.42.3.671) - R1_run_dnn_vgg16_places365_manipulations.m: computes the integration measure for images with targeted manipulations - R1_analyze_dnn_places365_manipulations.m: relates the DNN measures for manipulated images to the behavioral ratings - /fMRI_analysis/analyze_fmri.m: relates fmri-based measures to the behavioral ratings - /behavior/R1_VPA_complexityorder.m: Performs a variance decomposition analysis for the behavioral ratings in Study 3 %%%%% Supplementary analysis scripts Note: These scripts needs to be moved to the top-level analysis folder to run them Category-analysis based on k-means clustering: - R1_classify_dnn_vgg16_places365.m: Extract outputs from the last FC layer, used for dividing the image sets into categories - R1_analyze_dnn_places365_categories.m: Compute results for different categories based on k-means clustering Symmetry analysis: - R1_compute_symmetry.m: compute horizontal / vertical symmetry for all images - R1_analyze_dnn_places365_symmetry.m: Compute results for the symmatry analysis Random splits on the 4x4 and 8x8 grid: - R1_run_dnn_vgg16_places365_randomsplits4x4.m: Extract DNN measures for the randomsplits in a the 4x4 grid - R1_run_dnn_vgg16_places365_randomsplits8x8.m: Extract DNN measures for the randomsplits in a the 8x8 grid - R1_analyze_dnn_places365_randomsplits4x4.m: Compute results for the randomsplits in a the 4x4 grid - R1_analyze_dnn_places365_randomsplits8x8.m: Compute results for the randomsplits in a the 8x8 grid Variance partitioning analysis: - R1_analyze_VPA_dnn_places365.m: Compute results of the variance partitioning Predictions based on a linear model trained on all DNN features: - R1_run_dnn_vgg16_places365_regression.m: Compute results for predicted versus real beauty ratings %%%%% Helper functions Helper functions are in the folder /helper_functions/, which should be added to the matlab path to run some of the scripts. They include: - VPA2.m: a variance partitioning analysis for the case where a variable is predicted by 2 predictors - VPA3.m: a variance partitioning analysis for the case where a variable is predicted by 3 predictors - fdr_bh.m: performs FDR correction on an array of p-values - fastpca.m: an more efficient PCA implementation needed for the prediction-based analysis - SHINE_luminance.m: Luminance matching using the SHINE toolbox - SHINE_histogram.m: Luminance&histogram matching using the SHINE toolbox