OpenAI Codex: Difference between revisions

Based on GPT-3, a [[neural network]] trained on text, Codex has additionally been trained on 159 gigabytes of [[Python (programming language)|Python]] code from 54 million [[GitHub]] repositories.<ref name="VB-bias">{{Cite news|last=Wiggers|first=Kyle|date=July 8, 2021|title=OpenAI warns AI behind GitHub’s Copilot may be susceptible to bias|work=[[VentureBeat]]|url=https://venturebeat.com/2021/07/08/openai-warns-ai-behind-githubs-copilot-may-be-susceptible-to-bias/|access-date=2021-09-03}}</ref><ref name="IQ">{{Cite news|last=Alford|first=Anthony|date=August 31, 2021|title=OpenAI Announces 12 Billion Parameter Code-Generation AI Codex|work=InfoQ|url=https://www.infoq.com/news/2021/08/openai-codex/|access-date=2021-09-03}}</ref> A typical use case of Codex is typing a comment, such as "<code>//compute the moving average of an array for a given window size</code>", then using the AI to suggest a block of code satisfying that prompt.<ref name="RegTA">{{Cite news|last=Anderson|first=Tim|last2=Quach|first2=Katyanna|date=July 6, 2021|title=GitHub Copilot auto-coder snags emerge, from seemingly spilled secrets to bad code, but some love it|work=[[The Register]]|url=https://www.theregister.com/2021/07/06/github_copilot_autocoder_caught_spilling/|access-date=2021-09-04}}</ref> OpenAI has stated that Codex can complete approximately 37% of requests and is meant to make human programming faster rather than replace it; according to OpenAI's blog, Codex excels most at "mapping [...] simple problems to existing code", which they describe as "probably the least fun part of programming".<ref name="SH">{{Cite news|last=Dorrier|first=Jason|date=August 15, 2021|title=OpenAI’s Codex Translates Everyday Language Into Computer Code|work=[[SingularityHub]]|url=https://singularityhub.com/2021/08/15/openais-codex-translates-everyday-language-into-computer-code/|access-date=2021-09-03}}</ref><ref name="VB">{{Cite news|last=Dickson|first=Ben|date=August 16, 2021|title=What to expect from OpenAI’s Codex API|work=[[VentureBeat]]|url=https://venturebeat.com/2021/08/16/what-to-expect-from-openais-codex-api/|access-date=2021-09-03}}</ref> [[Jeremy Howard (entrepreneur)|Jeremy Howard]], co-founder of [[Fast.ai]], stated that "[Codex] is a way of getting code written without having to write as much code" and that "it is not always correct, but it is just close enough".<ref name="NYT">{{Cite news|last=Metz|first=Cade|date=September 9, 2021|title=A.I. Can Now Write Its Own Computer Code. That’s Good News for Humans.|work=[[The New York Times]]|url=https://www.nytimes.com/2021/09/09/technology/codex-artificial-intelligence-coding.html|access-date=2021-09-16}}</ref> According to a paper written by OpenAI researchers, when attempting each test case 100 times, 70.2% of prompts had working solutions.<ref name="arXiv">{{Cite arxiv|last=Chen|first=Mark|last2=Tworek|first2=Jerry|last3=Jun|first3=Heewoo|last4=Yuan|first4=Qiming|last5=Pinto|first5=Henrique Ponde de Oliveira|last6=Kaplan|first6=Jared|last7=Edwards|first7=Harri|last8=Burda|first8=Yuri|last9=Joseph|first9=Nicholas|last10=Brockman|first10=Greg|last11=Ray|first11=Alex|date=2021-07-14|title=Evaluating Large Language Models Trained on Code |arxiv=2107.03374 |class=cs}}</ref>

 

 

 

== Issues ==

OpenAI demonstrations showcased flaws such as inefficient code and one-off quirks in code samples.<ref name="VB" /> In an interview with ''[[The Verge]]'', OpenAI [[chief technology officer]] Greg Brockman said that "sometimes [Codex] doesn't quite know exactly what you're asking" and that it can require some trial and error.<ref name="Verge" /> OpenAI researchers found that Codex struggles with multi-step and higher-level prompts, often failing or yielding counter-intuitive behavior. Additionally, they brought up several safety issues, such as over-reliance by novice programmers, biases based on the training data, and security impacts due to vulnerable code.<ref name="arXiv" />

 

 

The [[Free Software Foundation]] has expressed concerns that code snippets generated by Copilot and Codex could unknowingly violate the terms of [[free software licenses]], such as the [[GPL]], which requires derivative works to be licensed under equivalent terms.<ref name="IW-FSF">{{Cite news|last=Krill|first=Paul|date=August 2, 2021|title=GitHub Copilot is ‘unacceptable and unjust,’ says Free Software Foundation|work=[[InfoWorld]]|url=https://www.infoworld.com/article/3627319/github-copilot-is-unacceptable-and-unjust-says-free-software-foundation.html|access-date=2021-09-03}}</ref> Issues they raised include whether training on public repositories falls into [[fair use]] or not, how developers could discover infringing generated code, whether trained [[machine learning]] models could be considered modifiable source code or a compilation of the training data, and if machine learning models could themselves be copyrighted and by whom.<ref name="IW-FSF" /><ref name="FSF">{{Cite news|last=Robertson|first=Donald|date=2021-07-28|title=FSF-funded call for white papers on philosophical and legal questions around Copilot: Submit before Monday, August 23, 2021|work=[[Free Software Foundation]]|url=https://www.fsf.org/blogs/licensing/fsf-funded-call-for-white-papers-on-philosophical-and-legal-questions-around-copilot|access-date=2021-09-04}}</ref> An internal GitHub study found that approximately 0.1% of generated code contained direct copies from the training data. One specific example has been raised, in which the model outputted the original code of the [[fast inverse square root]] algorithm, including comments and an incorrect copyright notice.<ref name="RegTA"/>