Model Editing for LLMs4Code: How Far are We?

CNLE and CSNE datasets can be found in datasets.zip. The source code is available at: https://github.com/xpq-tech/code-llmedit.git.

The prompt templates used to construct datasets are as follows:

• Prompt template to evaluate the semantic matches between the code and the description:

I'm filtering a python code dataset, and I want to get high-quality data from it.

I will give you an Intent that represents the intention for generating code and a Snippet that is a fragment of the desired code.

Please rate the degree of match between the Intent and the Snippet, with scores ranging from 0 to 4. The scoring rules are as follows:

0: Very Irrelevant.

1: Irrelevant.

2: Neutral.

3: Relevant.

4: Very Relevant.

# EXAMPLES #

# 1

Intent: sort a nested list l by two elements '1' and '0'

Snippet: sorted(l, key=lambda x: (-int(x[1]), x[0]))

Score:4

# 2

Intent: make curvilinear plots in matplotlib

Snippet: plt.show()

Score:1

# RESPONSE #

Response a score between 0 to 4 with no more explanations.

# INSTANCE #

Intent: {}

Snippet: {}

Score:

• Prompt template to generate rewritten intents:

You need to rephrase an intent statement (i.e., 'Intent') that generates Python code (i.e., 'Snippet') to create a new sentence (i.e., 'Rephrased Intent'), ensuring that this sentence is different from the original intent statement but conveys the same semantics.

The intention statement and the new statement are both aimed at generating code snippets.

# EXAMPLE #

Snippet: sorted(l, key=lambda x: (-int(x[1]), x[0]))

Intent: sort a nested list l by two elements '1' and '0'

Rephrased Intent: rearrange the nested list 'l' considering two elements, '1' and '0'

# INSTANCE #

Snippet: {}

Intent: {}

Rephrased Intent: