Whodunit: Classifying Code as Human Authored or GPT-4 Generated - A case study on CodeChef problems

Artificial intelligence (AI) assistants such as GitHub Copilot and ChatGPT, built on large language models like GPT-4, are revolutionizing how programming tasks are performed, raising questions about whether code is authored by generative AI models. Such questions are of particular interest to educators, who worry that these tools enable a new form of academic dishonesty, in which students submit AI generated code as their own work. Our research explores the viability of using code stylometry and machine learning to distinguish between GPT-4 generated and human-authored code. Our dataset comprises human-authored solutions from CodeChef and AI-authored solutions generated by GPT-4. Our classifier outperforms baselines, with an F1-score and AUC-ROC score of 0.91. A variant of our classifier that excludes gameable features (e.g., empty lines, whitespace) still performs well with an F1-score and AUC-ROC score of 0.89. We also evaluated our classifier with respect to the difficulty of the programming problem and found that there was almost no difference between easier and intermediate problems, and the classifier performed only slightly worse on harder problems. Our study shows that code stylometry is a promising approach for distinguishing between GPT-4 generated code and human-authored code.
 

# Whodunit: CodeChef AI & Human Solutions Dataset

This repository contains the data for the [CodeChef](https://www.codechef.com/) problems and the code used to collect, extract code-style and code complexity features from it, as well as the code to build and evaluate the classifiers for the paper `Whodunit: Classifying Code as Human Authored or GPT-4 Generated - A case study on CodeChef problems`. It also contains the modified baseline code.
 

## Data Collection

The data corresponds to `399` problems filtered from the initial set of `1100` problems. The code used for collection and filtering can be found in the scripts directory. The final data is stored as described below.

### Files 

- `final_dataset.json`: Contains the data for all the problems on CodeChef.

- `final_successful_dataset.json`: Contains the same data as `final_dataset.json` but the `ai_solutions` field only contains the solutions that successfully passed the public test cases.

The `final_files` directory contains zipped sets of python files (binned into `easy`, `medium` and `hard` difficulties) that were used for the experiments in the paper.

- `unverified.zip`: Contains 2 Human and 2 AI solutions for each problem.

- `verified.zip`: Contains solutions that are verified to pass the public test cases.

- `sampled.zip`: Contains a random set of problems sources from the `unverified` set such that it matches the distribution of the `verified` set.
 

### Data Format

The JSON data is stored as a nested dictionary. The top level keys are the `11 difficulty levels` of CodeChef. Each difficulty level is a dictionary with the key being the `problem_code_id` on CodeChef and the value being the data for the problem. The data for each problem is structured in a dictionary with the following keys: 

- `constraints`: A string describing any constraints related to the problem.

- `subtasks`: A string detailing the subtasks associated with the problem.

- `sample_test_cases`: An array of dictionaries, each representing a public test case. Each test case includes:

- `input`: The input given to the problem.

- `output`: The expected output for the given input.

- `explanation`: A detailed explanation of why the output is as expected.

- `problem_statement`: A string describing the problem, its background, and requirements.

- `input_format`: A string describing the format in which input is provided.

- `output_format`: A string describing the format in which output is expected.

- `problem_name`: The name of the problem.

- `user_tags`: An array of strings representing user-defined tags for the problem.

- `computed_tags`: An array of strings representing system-generated tags for the problem.

- `problem_code_id`: A string representing the unique code ID of the problem.

- `difficulty_level`: A string or number indicating the difficulty level of the problem.

- `ai_solutions`: An array of strings, each representing GPT-4 (v0613) generated solution to the problem.

- `human_solutions`: An array of dictionaries, each containing details about a solution submitted by a user, which includes:

- `id`: A unique identifier for the solution.

- `submission_date`: The date of submission.

- `language`: The programming language used.

- `username`: The username of the submitter.

- `user_rating_star`: The user's rating.

- `contest_code`: The code of the contest in which the solution was submitted.

- `tooltip`: Status of the solution (e.g., accepted, rejected).

- `score`: The score achieved by the solution.

- `points`: The points achieved by the solution.

- `icon`: A link to an icon representing the status of the solution.

- `time`: The execution time of the solution.

- `memory`: The memory used by the solution.

- `solution`: A unique identifier for the solution.

- `code`: The actual code of the solution. 

**Note:** The `input_format`, `output_format` and `constraints` fields are not available for older problems on CodeChef. In such cases, the information is present in the `problem_statement` field.

## Feature Extraction

Before extracting the features, the comments and multi-line strings are to be removed by running the `scripts/remove_all_comments.py` script. This script accepts the `source_directory`, `destination_directory` and `output_file_path` which are the paths to the directory containing the files, the directory to store the files with comments removed and the path to a file that logs information about the file and removal process.

The code for extracting the code-style and code complexity features from the dataset can be found in the `script/feature_extraction` directory. This directory has 3 files: `extract_main_features.py`, `extract_non_gameable_features.ipynb`, and `extract_with_halstead_features.ipynb` corresponding to the 3 different feature sets used in the paper. These scripts read the files from the input path provided and outputs the extracted features to the output path provided. The extracted features are stored in the `data/features` directory from which 8 features files are generated. The features files are named as follows:

- `main_features.csv`: Contains the main classifier's features (`RQ1`).

- `with_halstead_features.csv`: Contains the halstead features (`RQ1`).

- `non_gameable_features.csv`: Contains the non-gameable features (`RQ2`).

- `correct_solutions_features.csv`: Contains the features for solutions that passed the public test cases (`RQ3`).

- `sampled_solutions_features.csv`: Contains the features for solutions sampled from the unverified set (`RQ3`).

- `easy_problems_features.csv`: Contains the features for solutions to the easy problems (`RQ4`).

- `medium_problems_features.csv`: Contains the features for solutions to the intermediate problems (`RQ4`).

- `hard_problems_features.csv`: Contains the features for solutions to the hard problems (`RQ4`).

**Note:** The extraction scripts require that the directory to the respective dataset is provided as input and the output path is specified. The output path in this package is the `data/features` directory.

## Classification

There are 8 scripts corresponding to the 8 different classifiers used in the paper and the 8 extracted features files. The scripts are found in the `scripts/classification` directory. The scripts are named as follows:

- `main_classification.py`: Contains the code for the main classifier (`RQ1`).

- `with_halstead_classification.py`: Contains the code for the halstead classifier (`RQ1`).

- `non_gameable_classification.py`: Contains the code for the non-gameable classifier (`RQ2`).

- `correct_solutions_classification.py`: Contains the code for the classifier trained on solutions that passed the public test cases (`RQ3`).

- `sampled_solutions_classification.py`: Contains the code for the classifier trained on solutions sampled from the unverified set (`RQ3`).

- `easy_problems_classification.py`: Contains the code for the classifier trained on solutions to the easy problems (`RQ4`).

- `medium_problems_classification.py`: Contains the code for the classifier trained on solutions to the intermediate problems (`RQ4`).

- `hard_problems_classification.py`: Contains the code for the classifier trained on solutions to the hard problems (`RQ4`).

Each of these scripts accept a `.csv` file containing the features as input and prints out the results. The scripts also display the SHAP plot for the top 10 features.

## For more information, please refer to the `README.md file`