Replication package for the paper: "Datasets, Bias, Licenses, and Terms of Use: A Large and Longitudinal Study on the Documentation of Hugging Face Machine Learning Models"

This replication package contains datasets and scripts related to the paper: "Datasets, Bias, Licenses, and Terms of Use: A Large and Longitudinal Study on the Documentation of Hugging Face Machine Learning Models"

root directory

• statistics.r: R script used to compute the correlation between usage and downloads, and the RQ1/RQ2 inter-rater agreements

scripts/

Directory containing all the scripts used to collect and process data. For further details, see the README file inside the script directory.

further-analyses/

Directory containing scripts and data used for further analyses. For further details, see the README file inside the directory.

first-snaphot

Contains data related to the first snapshot (ICPC 2024 study)

• modelsInfo.zip: zip file containing all the downloaded model cards (in JSON format)

Dataset

Dataset/DatasetHFmodellist.csv: list of HF models analyzed, with the following information: id,downloads,likes,tags,pipelinetag,pipelinecategory,License,licensemodelpermissivity Dataset/DatasetGitHubprjlistTransformers.txt: list of GitHub projects using the transformers library Dataset/DatasetGitHubprjlistDiffusers.txt: list of GitHub projects using the diffuserslibrary Dataset/DatasetGitHubprjfrompretrainedTransformers.txt: list of GitHub projects using the "frompretrained" transformers library Dataset/DatasetGitHubprjfrompretrainedDiffusers.txt: list of GitHub projects using the "frompretrained" diffusers library Dataset/DatasetGitHubprjmodelusedTransformers.csv: contains usage pairs: project, model for transformers library Dataset/DatasetGitHubprjmodelusedDiffusers.csv: contains usage pairs: project, model for diffusers library Dataset/DatasetIntersectedModels.csv : contains the models shared between the first and second snapshot for category Dataset/modelsReadme: contains the model cards belonging to the sample size Dataset/projectswith5ormorestars.csv: contains the projects with numStars major of 5 Dataset/projectsstars_summary.csv: contains the number of total projects with numStars RQ1

RQ1/RQ1datasetlistHF.txt: list of HF datasets RQ1/RQ1datasetTags.txt: list of models declaring the dataset tag RQ1/RQ1modelDataset.csv : list of models declaring the dataset tag with their respective datasets RQ1/RQ1datasetSample.csv: sample set of models used for the manual analysis of datasets RQ2

RQ2/RQ2biasclassification_sheet.csv: results of the manual labeling RQ3

RQ3/RQ3LicenseModels.csv: model license list,categorized by permissiveness, with the respective number of occurrences RQ3/RQ3LicenseprjTransformers.csv: transformers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3LicenseprjDiffusers.csv : diffusers project license list, categorized by permissiveness, with the respective number of occurrences RQ3/RQ3prjmodellicensepermissivityTransformersDiffusers.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffuserslibrary RQ3/RQ3prjmodellicensepermissivityTransformersDiffusersStarmajor5.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library for numStar > 5 RQ3/RQ3ContingencyMatrixpermissivityTransformersDiffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness RQ3/RQ3ContingencyMatrixlicensesTransformersDiffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffuserslibrary in terms of licenses RQ3/RQ3ContingencyMatrixpermissivityTransformersDiffusersStarmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness for projects with numStar > 5 RQ3/RQ3ContingencyMatrixlicensesTransformersDiffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses for projects with numStar > 5 RQ4

RQ4/RQ4TermsofUseSnapshot1.csv: results of the manual labeling related to terms of use for the first snapshot RQ4/RQ4TermsofUseSnapshot2.csv: results of the manual labeling related to terms of use for the second snapshot

Dataset

• Dataset/Dataset_HF-models-list.csv: list of HF models analyzed
• Dataset/Dataset_github-prj-list.txt: list of GitHub projects using the transformers library
• Dataset/Dataset_github-Prj_model-Used.csv: contains usage pairs: project, model
• Dataset/Dataset_prj-num-models-reused.csv: number of models used by each GitHub project
• Dataset/Dataset_model-download_num-prj_correlation.csv contains, for each model used by GitHub projects: the name, the task, the number of reusing projects, and the number of downloads

RQ1

• RQ1/RQ1_dataset-list.txt: list of HF datasets
• RQ1/RQ1_datasetSample.csv: sample set of models used for the manual analysis of datasets
• RQ1/RQ1_analyzeDatasetTags.py: Python script to analyze model tags for the presence of datasets. it requires to unzip the modelsInfo.zip in a directory with the same name (modelsInfo) at the root of the replication package folder. Produces the output to stdout. To redirect in a file fo be analyzed by the RQ2/countDataset.py script
• RQ1/RQ1_countDataset.py: given the output of RQ2/analyzeDatasetTags.py (passed as argument) produces, for each model, a list of Booleans indicating whether (i) the model only declares HF datasets, (ii) the model only declares external datasets, (iii) the model declares both, and (iv) the model is part of the sample for the manual analysis
• RQ1/RQ1_datasetTags.csv: output of RQ2/analyzeDatasetTags.py
• RQ1/RQ1_dataset_usage_count.csv: output of RQ2/countDataset.py

RQ2

• RQ2/tableBias.pdf: table detailing the number of occurrences of different types of bias by model Task
• RQ2/RQ2_bias_classification_sheet.csv: results of the manual labeling
• RQ2/RQ2_isBiased.csv: file to compute the inter-rater agreement of whether or not a model documents Bias
• RQ2/RQ2_biasAgrLabels.csv: file to compute the inter-rater agreement related to bias categories
• RQ2/RQ2_final_bias_categories_with_levels.csv: for each model in the sample, this file lists (i) the bias leaf category, (ii) the first-level category, and (iii) the intermediate category

RQ3

• RQ3/RQ3_LicenseValidation.csv: manual validation of a sample of licenses
• RQ3/RQ3_{NETWORK-RESTRICTIVE|RESTRICTIVE|WEAK-RESTRICTIVE|PERMISSIVE}-license-list.txt: lists of licenses with different permissiveness
• RQ3/RQ3_prjs_license.csv: for each project linked to models, among other fields it indicates the license tag and name
• RQ3/RQ3_models_license.csv: for each model, indicates among other pieces of info, whether the model has a license, and if yes what kind of license
• RQ3/RQ3_model-prj-license_contingency_table.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows)
• RQ3/RQ3_models_prjs_licenses_with_type.csv: pairs project-model, with their respective licenses and permissiveness level
• RQ3/license-task-relation/: contains scripts and data to perform the analysis between task types and license types:
  • analysis\_by\_type.py: script to perform the analysis
  • list.csv: the list of licenses with their permissiveness
  • licensebytype.csv: licenses of all analyzed models

RQ4

• RQ4/RQ4_Terms_of_Use_Snapshot1.csv: results of the manual labeling related to terms of use for the first snapshot

second-snaphot

Contains data related to the second snapshot.

Dataset

• Dataset/Dataset_HF_model_list.csv: list of HF models analyzed, with the following information: id,downloads,likes,tags,pipeline_tag,pipeline_category,License,license_model_permissivity
• Dataset/Dataset_GitHub_prj_list_Transformers.txt: list of GitHub projects using the transformers library
• Dataset/Dataset_GitHub_prj_list_Diffusers.txt: list of GitHub projects using the diffusers library
• Dataset/Dataset_GitHub_prj_frompretrained_Transformers.txt: list of GitHub projects using the "from_pretrained" transformers library
• Dataset/Dataset_GitHub_prj_frompretrained_Diffusers.txt: list of GitHub projects using the "from_pretrained" diffusers library
• Dataset/Dataset_GitHub_prj_model_used_Transformers.csv: contains usage pairs: project, model for transformerslibrary
• Dataset/Dataset_GitHub_prj_model_used_Diffusers.csv: contains usage pairs: project, model for diffusers library
• Dataset/Dataset_IntersectedModels.csv : contains the models shared between the first and second snapshot for category
• Dataset/modelsReadme: contains the model cards belonging to the sample size
• Dataset/projects_with_5_or_more_stars.csv: contains the projects with numStars major of 5
• Dataset/projects_stars_summary.csv: contains the number of total projects with numStars

RQ1

• RQ1/RQ1_dataset_list_HF.txt: list of HF datasets
• RQ1/RQ1_datasetTags.txt: list of models declaring the dataset tag
• RQ1/RQ1_modelDataset.csv : list of models declaring the dataset tag with their respective datasets
• RQ1/RQ1_datasetSample.csv: sample set of models used for the manual analysis of datasets

RQ2

• RQ2/RQ2_bias_classification_sheet.csv: results of the manual labeling

RQ3

• RQ3/RQ3_License_Models.csv: model license list,categorized by permissiveness, with the respective number of occurrences
• RQ3/RQ3_License_prjTransformers.csv: transformers project license list, categorized by permissiveness, with the respective number of occurrences
• RQ3/RQ3_License_prjDiffusers.csv : diffusers project license list, categorized by permissiveness, with the respective number of occurrences
• RQ3/RQ3_prj_model_license_permissivity_Transformers_Diffusers.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library
• RQ3/RQ3_prj_model_license_permissivity_Transformers_Diffusers_Starmajor5.csv: total list of projects that reuse the models, with their respective licenses and permissiveness related to Transformers and Diffusers library for numStar > 5
• RQ3/RQ3_Contingency_Matrix_permissivity_Transformers_Diffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness
• RQ3/RQ3_Contingency_Matrix_licenses_Transformers_Diffusers.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses
• RQ3/RQ3_Contingency_Matrix_permissivity_Transformers_Diffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of permissiveness for projects with numStar > 5
• RQ3/RQ3_Contingency_Matrix_licenses_Transformers_Diffusers_Starmajor5.csv: usage contingency table between projects' licenses (columns) and models' licenses (rows) related to Transformers and Diffusers library in terms of licenses for projects with numStar > 5
• RQ3/license-task-relation/: contains scripts and data to perform the analysis between task types and license types:
  • analysis\_by\_type.py: script to perform the analysis
  • list.csv: the list of licenses with their permissiveness
  • licensebytype.csv: licenses of all analyzed models

RQ4

• RQ4/RQ4_Terms_of_Use_Snapshot2.csv: results of the manual labeling related to terms of use for the second snapshot