GitHub Copilot research recitation

Introduction

GitHub Copilot is trained on billions of lines of public code. The suggestions it makes to you are adapted to your code, but the processing behind it is ultimately informed by code written by others.

How direct is the relationship between the suggested code and the code that informed it? In a recent thought-provoking paper¹, Bender, Gebru et al. coined the phrase “stochastic parrots” for artificial intelligence systems, like the ones that power GitHub Copilot. Or, as a fellow machine learning engineer at GitHub² remarked during a water cooler chat: these systems can feel like “a toddler with a photographic memory.”

These are deliberate oversimplifications. Many GitHub Copilot suggestions feel specifically tailored to the particular code base the user is working on. Often, it looks less like a parrot and more like a crow building novel tools out of small blocks³. Yet there’s no denying that GitHub Copilot has an impressive memory: