Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions · HackerPapers discussions

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — mentioned by elil17

Sun, 08 Jan 2023 20:41:03 GMT

It's an anecdote which is meant to illustrate a widely observed behavior among many humans: we're better at accounting for frequent errors than we are at accounting for rare ones. This is well established in the field of human reliability analysis (HRA). Take a look at the HRA method HEART (https://www.epd.gov.hk/eia/register/report/eiareport/eia_224...). There are three different error producing conditions (EPCs) related to rare failures (EPCs 1, 3, and 12 in table 1.3). As for my argument that people who frequently work with this technology aren't going to be so susceptible to…

Paper and discussion history. A mention is not an endorsement.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — mentioned by moyix

Mon, 26 Dec 2022 20:52:00 GMT

This is a really nice study! It is very cool that they were able to get professional programmers to participate, this is something that is really hard to set up as an academic team. And yes, 47 participants is a small number, but apparently sufficient in this case to detect the effect (as evidenced by the p-values). It also lines up nicely with work we did last year, which looked at the security of Copilot's completions without any humans in the loop [1] and found that something like 40% of its completions in security sensitive scenarios were vulnerable. One thing I'm not sure of is how well…

Paper and discussion history. A mention is not an endorsement.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — mentioned by abirch

Sat, 25 Jun 2022 16:42:13 GMT

To be honest, I wish that better code repositories we're given more weight. Microsoft's should be given more weight, especially considering the security issues that is autosuggested by copilot. https://arxiv.org/abs/2108.09293

Paper and discussion history. A mention is not an endorsement.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — mentioned by moyix

Wed, 27 Oct 2021 22:00:29 GMT

This attack has been studied! https://arxiv.org/abs/2007.02220 Although our own work shows Copilot is pretty good at adding security flaws on its own: https://arxiv.org/abs/2108.09293

Paper and discussion history. A mention is not an endorsement.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — mentioned by mzs

Fri, 03 Sep 2021 14:54:47 GMT

the actual paper: https://arxiv.org/abs/2108.09293 previous discussion including comments from lead author: https://news.ycombinator.com/item?id=28279365

Paper and discussion history. A mention is not an endorsement.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — mentioned by wcarss

Fri, 03 Sep 2021 12:38:01 GMT

I couldn't find a link to the actual study anywhere in the article: https://arxiv.org/abs/2108.09293

Paper and discussion history. A mention is not an endorsement.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions — HN submission

Mon, 23 Aug 2021 17:41:06 GMT

An empirical cybersecurity evaluation of GitHub Copilot's code contributions

137 points; 42 discussion replies on HN. Paper and source history. A submission is not an endorsement.