Using AI to help write code? The tech just isn't there yet, says a company that tried
Kite, an early entrant into the code autocompletion developer tool space, has decided to call it a day, saying there is still a long way to go for the technology because the state of the art for ML on code is just not good enough yet.
The San Francisco startup has been working on Python and JavaScript auto code-completion tools for developers since 2014, aiming to reduce repetitive tasks in programming. Kite CEO and founder Adam Smith announced the end of Kite in a blogpost. He said Kite had grown to 500,000 active developers monthly, but that the company failed to monetize the product.
Artificial Intelligence
"We failed to deliver our vision of AI-assisted programming because we were 10+ years too early to market, i.e. the tech is not ready yet," he wrote. "We failed to build a business because our product did not monetize, and it took too long to figure that out."
Also: Tech jobs are changing. Here are the real skills you'll need to get promoted
Smith also suggests Kite could have, maybe, solved the issue of synthesizing code reliably, but that would have required $100 million and more engineers.
"The largest issue is that state-of-the-art models don't understand the structure of code, such as non-local context. We made some progress towards better models for code, but the problem is very engineering intensive. It may cost over $100 million to build a production-quality tool capable of synthesizing code reliably, and nobody has tried that quite yet," said Smith.
Kite showed promise as an alternative to Microsoft's code-completion tool IntelliSense, and in 2019, when it still only supported code completion in Python, it raised $19 million in a Series A round led by Trinity Ventures, with personal participation by then-new GitHub CEO Nat Friedman, as TechCrunch reported at the time.
But by 2020, with new competitors like TabNine using OpenAI's GPT-2 large language model (LLM) for language-agnostic code completion, Kite opted to redesign its product using GPT-2 to support autocomplete for 11 more languages. Eventually, Kite would also build an integration with Microsoft's popular VS Code editor.
Also: PepsiCo is working with startups to tap new sources of innovation. Here's how it does it
Kite has open sourced most of its code on Github here. It includes Kite's data-driven Python type inference engine, Python public-package analyzer, desktop software, editor integrations, Github crawler and analyzer, and more.
In June 2021, Microsoft released a preview of a VS Code extension that tapped OpenAI Codex to create GitHub Copilot. In June 2022, GitHub Copilot became generally available for $10 per developer per month or $100 per user per year.
However, Smith said the state-of-the-art ML is still not good enough.
"We built the most-advanced AI for helping developers at the time, but it fell short of the 10× improvement required to break through because the state of the art for ML on code is not good enough. You can see this in Github Copilot, which is built by Github in collaboration with Open AI. As of late 2022, Copilot shows a lot of promise but still has a long way to go," he wrote.
However, Smith said that the future is still bright for these tools. "We can't wait for AI to revolutionize programming. Computers are so magical; it will be amazing to experience a step-function increase in what they can do for us," he said.