Generative AI has seized the popular imagination and started a new tech gold rush. While much attention has been focused on AI tools that produce natural language prose and visual art, in tech circles AI is gaining increased interest for its coding capabilities. You can describe a program you want to an AI chatbot, and it returns executable code to you within seconds, something that both intrigues and unnerves the average programmer.
The prospect of AI-driven programming has led to some fairly grandiose predictions about the future of the software industry, particularly from C-suite execs, consultants, and the pundits who follow them. But what about the programmers and managers who work with AI tools on a day-to-day basis? We asked a handful of people programming with generative AI how it's working out for them so far. What we learned is that AI really is changing the way people work—but machines aren't about to replace human coders anytime soon.
Table of Contents
How AI helps coders
The two generative AI tools most commonly used by developers we spoke to were ChatGPT — the widely known AI chatbot from OpenAI — and GitHub Copilot, which integrates into Visual Studio and other IDEs (integrated development environments). While both tools can generate code based on natural language queries, Copilot and its experimental successor, Copilot X, can go one step beyond the conversational model, acting as a sort of souped-up IDE auto-complete tool that anticipates what the developer is working on.
Vanessa Freudenberg, co-founder and chief architect at Croquet.io, says that she uses GitHub Copilot in her daily coding with Visual Studio Code. She explains how it works.
If I write the line:
let x = this.leftMargin + this.width / 2;
it will automatically suggest the next line:
let y = this.topMargin + this.height / 2;
And it knows that it needs to replace "width" and "left" with "height" and "top". That saves me a lot of typing.
Panickos Neophytou, co-founder and CTO at NetBeez, says he uses Copilot X and ChatGPT anytime he codes. He describes two different approaches to getting beyond auto-complete with these tools. The first is systematic. "Describe a very well-defined function with specific inputs, examples of expected outputs as well as the data models involved, which include database tables with implicit associations," he says. "AI can generally infer the associations. Ask it to implement it in a specific language and specific ways. The 'tasks' defined in project management tools should be defined as this kind of prompt."
Neophytou also thinks a more casual and conversational technique can produce good results. In this mode, he says, "while performing a task you ask questions that pop up in your mind about how to do certain things. This is like having an experienced engineer next to you answering your questions and guiding you towards finishing the task."
No matter what technique you use, learning to prompt the AI correctly is something of an art. "I use chain-of-thought prompting to ensure I get the correct verb to start and refine my prompt," says Shanea Leven, founder and CEO of software provider CodeSee. "Choosing the right verbs and being descriptive are really important for creating a good prompt." (Prompt engineering is becoming a hot discipline for this very reason.)
AI's coding strengths
The developers we talked to offered a variety of use cases where AI tools helped them get their work done. Here's what stood out.
Generating documentation from code
Software documentation is a fairly structured format, which is an area where AI thrives. "Something I love using ChatGPT for is writing documentation [or] comments in my code," says Chris Love, founder of web consultancy Love2Dev. "That takes so much time to do by hand. But once I have a function complete I can give it the function and have a nice page worth of documentation written in a second or two. I love that!"
Generating code from documentation
AI can also generate code based on comments or documentation. Croquet.io's Freudenberg gave a simple example of Copilot generating code from only one comment line, where the following comment:
// get file name from our url
produced this output:
let fileName = window.location.pathname.split("/").pop();
"I probably would have written this differently—I'm a regex girl—but it's a perfectly simple and arguably more readable solution, so I would just accept it," she says. "It's not only a huge time-saver, but sometimes it also teaches me idioms I may not have discovered myself."
Solutions to solved problems
Developers know that part of programming is constantly reinventing the wheel. It can be frustrating working on a problem that you know someone, somewhere, has already solved. Jeff Wills, engineering practice lead at software development firm Rise8, sees this as a domain ripe for AI assistance. "Let's say I make a method to calculate the distance between two points on a sphere," he explains. "Copilot will automatically go out and find the Haversine algorithm and generate all that code."
Wills finds this use case particularly handy in cases where the alternative to AI-generated code is adding a large library to his application. "Say I don't want to bring a whole geometry library into my code and bloat the codebase," he explains. "I really only need that one algorithm. I'll probably just write it myself and include it — or use ChatGPT or Copilot to help me produce that. That's the bread and butter for AI right now."
Updating or cleaning up code
Chris Love of Love2Dev finds ChatGPT particularly useful for updating code he's already written. "I have tackled updating old Node.js modules that I have not updated just because the time it takes to type the code is longer than it is worth," he says. "The most common situation I look for is converting older promise-based functions to use async/await. The latter is a cleaner syntax, but was not as common when I wrote the module. I also get it to use more modern syntaxes like destructuring and converting variable declarations from var to const and let."
Faster coding (maybe)
Many developers we spoke to said that working with Copilot or ChatGPT made them feel like they could do their work more quickly, though they admitted they couldn't necessarily quantify that. "What I think it does in the end is help me write better code a little faster," says Love. "It is difficult to say what percent faster, but it is tangible to me."
"I feel like I can iterate through possible solutions faster" says Rise8's Wills. "That should theoretically speed me up—but maybe I'm looking at more possible solutions! So maybe I don't get a payoff in time, but in quality, because I was able to iterate a little bit more."
What to watch out for
On the other hand, almost everyone we spoke to emphasized that AI tools aren't magic and there were plenty of pitfalls to look out for and reality checks you'd encounter when using them.
Be very specific
AI chatbots' natural language facilities can seem almost magical, but they probably don't understand as much as you might think (or hope) they do. You need to be as explicit and specific as possible to get the best results. "Currently, ChatGPT cannot see what you are coding in third-party apps you are using," says Liam Edwards, CEO of Liam's Professional Websites. "For this reason, you need to give it as much information on the coding issue or query you have. For example, if I said, 'Write me custom CSS that makes the button background white,' I would get code that may work or may not. If I said, 'Write me custom CSS code that makes the background of a button white (the button's class is .button)', I would get a much more accurate response for my situation."
"More value will come when it is possible to ask questions about a company’s specific codebase and then have the AI perform the actual task for you. That will make AI truly game-changing."
To Edwards, this exemplifies another important point about using these tools: You need to have coding experience and knowledge to get the most out of them. His example, he says, "shows how a person who does not code well would take longer and get less accurate responses."
Keep it simple
Peter Surowski, CTO of the web design and development agency Brain Jar, says he uses AI when he's writing code, "but it's only useful for small tasks, ones that you'd normally use Google for. For example, if I need some boilerplate code just to set things up, ChatGPT is great for that. Or if you just can't remember how to write a switch statement or a ternary function in whatever language, you can ask ChatGPT. But for anything more complicated, it's useless. I think the people who were saying it's going to take their jobs were being silly."
CodeSee's Leven says that "right now, ChatGPT is good at general code responses but not necessarily great at domain-specific code generation, scaffolding, and auto-complete. It’s great at searching but at the same time it’s limited mostly because of the small context size."
Still, she says, "I expect this to improve over time. More value will come when it is possible to ask questions about a company’s specific codebase and then have the AI perform the actual task for you. That will make AI truly game-changing."
Testing is a must
Most software shops have moved to a development regime where code needs to pass an extensive automated test suite before it goes into production. AI-generated code is no exception. While that may not be a drawback, per se, it may come as a disappointment to those with an unrealistic view of what the technology can achieve. "I have to review everything and of course test it," says Love2Dev's Love. "There are too many developers that have historically taken what is on Stack Overflow and accepted it as the right answer; I am sure the same is happening with ChatGPT."
A little tidying is necessary
Almost nobody told us they were cutting and pasting AI-generated code without modification. "Generally you have to make sure the code produced is correct and tested," says NetBeez's Neophytou. "Most of the time it needs to be refined a little—not majorly re-written. But in some rare cases, the code it produces is exactly what you needed, and those times are just mind-blowing and very rewarding (and at the same time scary)."
"Usually they'll have to work some style guide stuff to make it match the codebase," says Rise8's Wills. This work includes "renaming methods, changing the format a little bit, and keeping everything clean and consistent—like the codebase was written by one person."
Keep security in mind
Any new technique being used to write code is inevitably going to open up an attack surface to malicious actors in ways that are difficult to anticipate, and generative AI is no exception. In a recent example, security researchers found that ChatGPT hallucinations can include nonexistent npm packages — and that an attacker could potentially predict those package names, create them, and fill them with malicious code.
Surya Sanchez, founder of DeepIdea Lab, which uses AI extensively as part of its workflow, says that the way to fight such attacks is to "run the code locally, identify errors, and understand the AI-generated code is referring to non-existent packages. In those situations, we rewrite the code manually, providing clearer instructions to focus on particular sections instead of the entire code." Sanchez also advises that you "avoid sharing secrets or API keys, as AI could be reviewed by third parties. We want to ensure that sensitive information related to production remains secure."
In some ways, the current limitations of AI serve as a built-in security feature, at least for now. "The token or prompt/response size limitations sort of give you a guard against malicious code in my opinion," says Love. "It forces you to review everything in small chunks."
AI tools in schools and at work
As a teacher who works with computer programming students at the University of Amsterdam, Max van der Broek has a somewhat unique perspective on how up-and-coming programmers are thinking about AI. He recently conducted a survey that found that more than half of the students in his program were using ChatGPT for coding. (This is a higher percentage than those who use it for writing assignments.) One intriguing result from the survey is that the students want guidelines for ChatGPT use and find the current policy both unclear and too strict: They want some uses to be allowed, but not all. "I can imagine a future in which using generative AI is illegal in your first year as you learn the fundamentals, and it's allowed as a copilot in later years as you create bigger and more complex projects," van der Broek says, adding that "the best practices we have now will surely be outdated next year."
Developers in the workplace will also want to know how and when they can use AI on the job, so it's imperative that managers start figuring out the rules—because in all likelihood, some of them are doing it already. At Rise8, Wills says that use of AI was both a top-down and a bottom-up phenomenon. "Brian Kroger, our founder, was very interested," Wills says. "He likes to stay abreast of what's the most current technology, so he was posing questions in our engineering channels to get us thinking about it. But at the same time we had people that, when ChatGPT really blew up, were immediately out there as early adopters and started using that resource to see what it was producing and how they could integrate it into their daily workflow."
Wills also says that Rise8 has budgeted money that developers can use to pay for tools they find helpful. Many have chosen to pay for access to ChatGPT-4, which he says produces better results than the free ChatGPT-3.5.
What does the future hold?
One of the biggest anxieties that generative AI has given rise to is that it will eliminate scores of human jobs that seemed safe from automation—coding among them. While it's impossible to say what the future holds, the developers and managers we spoke to were mostly skeptical. "It can write chunks of code, and that's very impressive," says Brain Jar's Surowski. "But it can't implement it, turn it into a plugin, test it, fix problems. That's what a developer does. We're not just code writers. In fact, that's a fairly small part of our job. And it's the only part AI can help with."
Overall, CodeSee's Leven agrees. "I do think that there will be companies that choose to reduce the size of their engineering teams," she says, but "forward-thinking, best-in-class companies will not do this. They will keep their place to outpace their competitors and win their space. The companies that do this will still need good developers because the truth is that the AI can’t reason, weigh trade-offs, or handle anything nonlinear or complex yet—so for now, its help is limited unless you’re building something simple. We’ve seen the best outcomes from a number of companies that use humans and AI together."