Evaluating Language Model Agency through Negotiations

This repository contains the raw negotiation transcripts from the paper "Evaluating Language Model Agency through Negotiations" [1]. The data includes both transcripts from self-play (a model plays against itself; corresponding to Section 4.1 of paper) and cross-play (a model plays against another model; Section 4.2). In total, this encompasses 2926 transcripts (942 self-play, 1984 cross-play).

The transcripts are structured in the following way:

transcripts/

├── self_play/

│   └── {model_name}/

│       ├── processed_negotiation.csv

│       ├── negotiations.csv

│       ├── interrogations.csv

│       └── .hydra/

│           ├── config.yaml

│           ├── hydra.yaml

│           └── overrides.yaml

└── cross_play/

    └── {model_1_name}_{model_2_name}/

        ├── processed_negotiation.csv

        ├── negotiations.csv

        └── .hydra/

            ├── config.yaml

            ├── hydra.yaml

            └── overrides.yaml

We also include metadata with rules and game setup in the metadata folder.

metadata/

├── agents/

│   └── anon.yaml

└── game/

    ├── generic-rental-agreement.yaml

    ├── generic_game_rules.yaml

    └── issues/

        ├── {issue_i}

        ├── ...

Citation

Please cite the paper when using data.

[1] T.R. Davidson, V. Veselovsky, M. Josifoski, M. Peyrard, A. Bosselut, M. Kosinski, R. West (2023).

Evaluating Language Model Agency through Negotiations.