By: Albert Ziegler (@wunderalbert)
GitHub Copilot: Parrot or Crow?
A first look at rote learning in GitHub Copilot suggestions.
Introduction
GitHub Copilot is trained on billions of lines of public code. The suggestions it makes to you are adapted to your code, but the processing behind it is ultimately informed by code written by others.
How direct is the relationship between the suggested code and the code that informed it? In a recent thought-provoking paper1, Bender, Gebru et al. coined the phrase “stochastic parrots” for artificial intelligence systems like the ones that power GitHub Copilot. Or as a fellow machine learning engineer at GitHub2 remarked during a water cooler chat: these systems can feel like ”a toddler with a photographic memory.”
These are deliberate oversimplifications. Many GitHub Copilot suggestions feel pretty specifically tailored to the particular code base the user is working on. Often, it looks less like a parrot and more like a crow building novel tools out of small blocks3. But there’s no denying that GitHub Copilot has an impressive memory:
Here, I intentionally directed4 GitHub Copilot to recite a well known text it obviously knows by heart. I, too, know a couple of texts by heart. For example, I still remember some poems I learnt in school. Yet no matter the topic, not once have I been tempted to derail a conversation by falling into iambic tetrameter and waxing about daffodils.
So is that (or rather the coding equivalent of it) something GitHub Copilot is prone to doing? How many of its suggestions are unique, and how often does it just parrot some likely looking code it has seen during training?
The Experiment
During GitHub Copilot’s early development, nearly 300 employees used it in their daily work as part of an internal trial. This trial provided a good dataset to test for recitation. I wanted to find out how often GitHub Copilot gave them a suggestion that was quoted from something it had seen before.
I limited the investigation to Python suggestions with a cutoff on May 7, 2021 (the day we started extracting that data). That left 453,780 suggestions spread out over 396 “user weeks”, i.e. calendar weeks during which a user actively used GitHub Copilot on Python code.
Automatic Filtering
453,780 suggestions are a lot, but many of them can be dismissed immediately. To get to the interesting cases, consider sequences of “words” that occur in the suggestion in the same order as in the code GitHub Copilot has been trained on. In this context, punctuation, brackets, or other special characters all count as “words”, while tabs, spaces or even line breaks are ignored completely. After all, a quote is still a quote, whether it’s indented by 1 tab or 8 spaces.
For example, one of GitHub Copilot’s suggestions was the following regex for numbers separated by whitespace:
r'^\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+\s+\d+'
This would be exactly 100 “words” in the sense above, but it’s a particularly dense example: the average non-empty line of code has only 10 “words.” I’ve restricted this investigation to cases where the overlap with the code GitHub Copilot was trained on contains at least 60 such “words”. We have to set the cut somewhere, and I think it’s rather rare that shorter sequences would be of great interest. In fact, most of the interesting cases identified later are well clear of that threshold of 60.
If the overlap extends to what the user has already written, that also counts for the length. After all, the user may have written that context with the help of GitHub Copilot as well!
In the following example, the user has started writing a very common snippet. GitHub Copilot completes it. Even though the completion itself is rather short, together with the already existing code it clears the threshold and is retained.
This procedure is permissive enough to let many relatively “boring” examples through, like the two above. But it’s still effective at dialing in the human analysis to the interesting cases, sorting out over 99% of Copilot suggestions.
Manual Bucketing
After filtering, there were 473 suggestions left. But they came in very different forms:
- Some were basically just repeats of another case that passed filtering. For example, sometimes GitHub Copilot makes a suggestion, the developer types a comment line, and GitHub Copilot offers a very similar suggestion again. I removed these cases from the analysis as duplicates.
- Some were long, repetitive sequences. Like the following example, where the repeated blocks of
‘<p>’are of course found somewhere in the training set:
Such suggestions can be helpful (test cases, regexes) or not helpful (like this case, I suspect). But in any case, they do not fit the idea of rote learning I had in mind when I started this investigation. - Some were standard inventories, like the natural numbers, or the prime numbers, or stock market tickers, or the Greek alphabet:
- Some were common, straightforward ways, perhaps even universal ways, of doing things with very few natural degrees of freedom. For example, the middle part of the following strikes me as very much the standard way of using the BeautifulSoup package to parse a wikipedia list. In fact, the best matching snippet found in GitHub Copilot's training data5 uses such code to parse a different article and goes on to do different things with the results.
This doesn’t fit my idea of a quote either. It’s a bit like when someone says “I’m taking out the trash; I’ll be back soon” -- that’s a matter of fact statement, not a quote, even though that particular phrase has been uttered many times before. - And then there are all other cases. Those with at least some specific overlap in either code or comments. These are what interests me most, and what I’m going to concentrate on from now on.
This bucketing necessarily has some edge cases6, and your mileage may vary in how you think they should be classified. Maybe you even disagree with the whole set of buckets in the first place.
That’s why we’ve open sourced that dataset7. So if you feel a bit differently about the bucketing, or if you’re interested in other aspects of GitHub Copilot parroting its training set, you’re very welcome to just ignore my next section and draw your own conclusions.
Results
For most of GitHub Copilot's suggestions, our automatic filter didn’t find any significant overlap with the code used for training. But it