Github Copilot

This is an analysis of the Github Copilot extension for Visual Studio Code.

Under macOS the VS Code Extensions are located in the following directory:

~/.vscode/extensions

Analysis of version 1.92.177

For an analysis of Copilot Chat see README_COPILOT_CHAT.md.

Prompts

The Github Copilot extension generates three types of prompts.

Prompt 1: Single file

We start with the simplest case with only one file file1.py.

• filename: file1.py
• file content: # Print hello, world

If the user presses enter after # Print hello, world, the extension generates the following prompt:

# Path: file1.py
# Print hello, world

The path to the file is part of the prompt.

Prompt 2: Multiple files

Now let's consider a slightly more complex two-file case where file file2.py is edited.

• filename: file1.py

• file content: # Print hello, world

• filename: file2.py

• file content: # Print he

In this case, the extension generates the following prompt:

# Path: file2.py
# Compare this snippet from file1.py:
# # Print hello, world
# Print he

Files with similar content are also included in the prompt.

Prompt 3: Fill in the middle

Copilot supports Fill in the Middle. That means the extension sends the code before and after the cursor position to the model.

• filename: file3.py
• file content: # Test prefix\n# Test suffix

If the user presses enter after # Test prefix, the extension generates the prefix

# Path: file3.py
# Test prefix

and the suffix

# Test suffix

Communication

Language model

To generate a completion, the extension sends a POST request to the endpoint https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions.

After sending the request, the endpoint returns the following response.

Telemetry

The Github Copilot extension sends telemetry data to the endpoint https://dc.services.visualstudio.com:

Deeper Analysis

Vocabulary

The extension contains two vocabulary files

Filename	Vocabulary Size	Comment
vocab_cushman001.bpe	50,276	This vocabulary is based on the GPT-2 vocabulary
vocab_cushman002.bpe	100,000	This vocabulary is new and not based on the GPT-2 vocabulary anymore

Min prompt chars

The length of the prompt has to be >= 10 characters before the prompt is sent to the model.

if ((_ > 0 ? n.length : d) < t.MIN_PROMPT_CHARS)
    return t._contextTooShort;

File information

The following information is collected about the file being edited:

const m = {
    uri: d.toString(), // The absolute path of the file
    source: t, // Content of the file
    offset: n, // The offset of the cursor
    relativePath: u, // The relative path of the file
    languageId: p // The programming language of the file
}

Neighbor Files

The extension remembers the files that have been accessed before. The function getNeighborFiles calls the function truncateDocs. Input of the function truncateDocs are the files sorted by access time.

When the combined size of all files exceeds 200,000, any additional files will be disregarded. The function truncateDocs returns a truncated list of files.

Copilot Performance

We have evaluated the copilot model cushman-ml with the HumanEval dataset. Out of 164 programming problems, the model can solve 56.10%.

Model name	Pass@1	Date	Comment
code-cushman-001	32.93%	2022-10-23	https://openai.com/api/
code-davinci-002	46.95%	2022-10-23	https://openai.com/api/
cushman-ml	56.10%	2022-10-23	Copilot

Completions of the evaluation run: 2022-10-23-samples-cushman-ml.jsonl