{"id":2089,"date":"2021-07-22T07:24:55","date_gmt":"2021-07-22T14:24:55","guid":{"rendered":"https:\/\/coderpad.io\/?p=2089"},"modified":"2023-06-05T14:38:53","modified_gmt":"2023-06-05T21:38:53","slug":"github-copilot-breaks-bad-interviews","status":"publish","type":"post","link":"https:\/\/coderpad.io\/blog\/development\/github-copilot-breaks-bad-interviews\/","title":{"rendered":"GitHub Copilot Breaks Bad Interviews"},"content":{"rendered":"\n

GitHub Copilot<\/a> was recently announced! In the past few years, we\u2019ve seen artificial intelligence (AI) revolutionize aspects of technology such as image recognition, recommendation algorithms, and more. With Copilot, GitHub is hoping to add \u201ccode generation\u201d to that list of items. In this article, we\u2019ll explain what GitHub Copilot is, why some people are concerned with allowing its usage in technical interviews, and how to build your interviews to be better in general and therefore more resilient to auto-generated code.<\/p>\n\n\n

What is GitHub Copilot?<\/h2>\n\n

GitHub Copilot is a VSCode extension that\u2019s able to autocomplete code snippets. While we’ve had some form of code completion for some time now, they’ve been relatively limited in one way or another. Usually, they tend to rely on existing code in your files or can only suggest a tiny amount of code per suggestion. Other times, code completion can be no more than a predefined snippet of code without considering any additional context.<\/p>\n\n

Copilot offers the ability to suggest multi-line and contextually aware code suggestions – powered by AI. GitHub says this tool is available for most programming languages and can implement entire functions off of their names alone.<\/p>\n\n

They\u2019re not alone in this endeavor, however. GitHub is working in collaboration with OpenAI, the company behind the extremely powerful GPT-3 AI tool. Between the training dataset consisting of GitHub\u2019s public source code and OpenAI\u2019s expertise in AI generation, it has the potential to provide impressive results.<\/p>\n\n

Taking an example from their website, it\u2019s able to generate an implementation of a date utility from nothing more than the function name.<\/p>\n\n\n

\"\"<\/figure>\n\n\n

It\u2019s incredible that given the simple input of:<\/p>\n\n

function<\/span> calculateDaysBetweenDates<\/span>(date1, date2<\/span>) <\/span>{<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n
It\u2019s able to generate the following code which functions how you might expect:<\/p>\n\n
function<\/span> calculateDaysBetweenDates<\/span>(date1, date2<\/span>) <\/span>{\n  var<\/span> oneDay = 24<\/span> * 60<\/span> * 60<\/span> * 1000<\/span>;\n  var<\/span> date1InMillis = date1.getTime();\n  var<\/span> date2InMillis = date2.getTime();\n  var<\/span> days = Math<\/span>.round(Math<\/span>.abs(date2InMillis - date1InMillis) \/ oneDay);\n  return<\/span> days;\n}<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n
While every new tech has its naysayers, and even some on our team are skeptical of its true utility, it\u2019s undoubtedly a powerful tool that will, for some, change the way they write code. <\/p>\n\n\n
How does Copilot Threaten Tech Interviews?<\/h2>\n\n\n
In the current tech interview landscape, algorithms reign as the leading method of technical assessment. While we\u2019ve touched on why this shouldn\u2019t be the case<\/a> before, Copilot in particular breaks algorithm questions in significant ways by being able to rapidly generate solutions to algorithmic problems.<\/p>\n\n\n
Let\u2019s see how Copilot interacts with some common interview questions.<\/p>\n\n
One common question that\u2019s a favorite of interviewers hoping to quickly glean mathematical competency is a function to check if a given number is prime or not.

Well, it\u2019s been a few years since I\u2019ve refreshed my math skills, so they might be a little shakey. I should be able to figure it out all the same. <\/p>\n\n
Let\u2019s open VSCode and start implementing the function.<\/p>\n\n\n
function<\/span> isPrime<\/span>(n<\/span>) <\/span>{\n\u00a0 \u00a0 if<\/span> (n < 2<\/span>) return<\/span> false<\/span>;\n\u00a0 \u00a0 for<\/span> (var<\/span> i = 2<\/span>; i < n; i++) {\n\u00a0 \u00a0 \u00a0 \u00a0 if<\/span> (n % i == 0<\/span>) return<\/span> false<\/span>;\n\u00a0 \u00a0 }\n\u00a0 \u00a0 return<\/span> true<\/span>;\n}<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n
Sure, we could change the var<\/code> to a const<\/code>, but this is code that I wouldn\u2019t blink twice at in a code review!<\/p>\n\n
While this may seem like an outlier, Copilot seems to excel in these algorithm-based questions. <\/p>\n\n\n
But surely isPrime<\/code> would be too trivial for areal<\/em> interview question, right? Perhaps, but watch what happens when a popular coding YouTuber attempts to utilize Copilot to solve Leetcode interview questions<\/p>\n\n\n\n
\n\n  \n            Play Video: GitHub Copilot CRUSHES Leetcode Interview Questions! \ud83d\ude33<\/span>\n      <\/a>\n<\/lite-youtube>\n\n<\/div><\/figure>\n\n\n
Without fail, Copilot is able to generate usable code for each difficulty level of algorithms given to it. Further, the solutions are all more performant than the average of submissions. For the permutation question, it\u2019s able to be faster than 88% of other submissions!<\/p>\n\n
\u201cGiven how predominant these types of interview questions are, we could be in big trouble if some candidates are less-than-forthcoming with their usage of Copilot in coding exercises.\u201d

While this may be true for some interviews, if your interviews are susceptible to collapse with Copilot in the picture, you\u2019re already facing this problem whether you\u2019re aware of it or not.<\/p>\n\n
Algorithms Are Breaking Your Interviews<\/h2>\n\n
On paper, algorithm based interview questions sound like a great way to assess a candidate’s skills. They can help give guidance on a candidate’s understanding of logic complexity, how efficient (or inefficient) a specific solution is, and is usually an insight into a candidate\u2019s ability to think in abstract manners. <\/p>\n\n
However, in practice algorithm questions tend to go against the grain of real-world engineering. Ideally, an interview process should act as a way to evaluate a candidate\u2019s ability to do the same kind of engineering they\u2019d be using in their projects at your company. While a developer may, with resources, implement an algorithm once in a while, they\u2019re more than likely doing things like refactoring significantly more often.<\/p>\n\n\n
We\u2019ve written more about how algorithms aren\u2019t often effective as interview questions in the past.<\/a><\/p>\n\n\n
But more than being unrepresentative of the job, algorithm questions are often easy to cheat – without the use of GitHub Copilot. Because most algorithm questions are significantly similar to each other, there\u2019s often a small selection of tips and tricks a candidate can memorize in order to drastically improve their output in these styles of interviews.<\/p>\n\n

There\u2019s even the potential for a candidate to be able to output an algorithm verbatim. There are hundreds of sites that will give a candidate one algorithm question after another in the hopes of improving their understanding of these algorithms.<\/p>\n\n\n
But with GitHub Copilot, the propensity for cheating on an algorithm question rises significantly. As we\u2019ve demonstrated previously in the article, it\u2019s capable of generating significant portions of code at a time. In fact, Copilot is so proficient at algorithm questions that a non-trivial number of algorithm questions we asked it to solve were done<\/a> before we could even finish the function signature. All it takes is a candidate to give Copilot the name of the function and paste the results into their assessment editor.<\/p>\n\n\n\n
Further, folks wanting to cheat have had the ability to do something similar for some time now in the form of forum questions. Simply lookup any algorithm on a code forum or site like StackOverflow<\/a> and you can find hundreds of answers at your disposal. <\/p>\n\n\n\n
In fact, many have pointed out that Copilot\u2019s process of looking up code based on its expected constraints is similar to one that a developer might experience by searching StackOverflow for code snippets. Funnily, some thought the idea so similar they decided to build an alternative VSCode plugin to Copilot that simply looks up StackOverflow answers as suggestions<\/a>.<\/p>\n\n\n
Regardless of how the question is answered, a candidate\u2019s skill at these types of questions might not reflect their capabilities in other aspects of the codebase. That\u2019s why, in order to access candidates properly, it might be the right move to move away from these questions.<\/p>\n\n
How to Fix Your Interviews<\/h2>\n\n\n
While you could<\/em> simply require candidates to use a non-VSCode IDE for your technical assessments, there are no guarantees that your take-home projects will be spared the same fate. Further, while VSCode is the launch platform for Copilot, it\u2019s more than likely to gain plugins for other IDEs in the future as well.<\/p>\n\n\n\n
But what of it? Let\u2019s say your company doesn\u2019t do take-home projects (even though you totally should<\/a>), what of it? Well, even if restricting VSCode would work to avoid Copilot for a while, you ideally want to be able to standardize your IDE platform for all candidates. <\/p>\n\n\n
Plus, as we touched on in the previous section, algorithm-based interview questions are still able to be manipulated – with or without Copilot.<\/p>\n\n
The solution to fixing your interviews should be two-fold:<\/p>\n\n\n
  1. More representative questions of the job
GitHub\u2019s not alone in this venture of AI-powered code generation, either! Other companies, such as Kite<\/a> or TabNine<\/a> are working hard at this problem space as well! Whether we want it to or not, AI-generated code helpers are here to stay.<\/p>\n\n\n