Is prompt engineer displacing data scientist as the 'sexiest job of the 21st century'?
More than a decade ago, in a Harvard Business Review article, Thomas Davenport declared data scientist to be the "sexiest job of the 21st century". Today, in an age of generative artifical intelligence (AI), is "prompt engineer" about to assume that title?
What's already certain is that it's one of the hottest jobs around. Prompt engineering involves getting the best and most relevant answers from generative AI tools, and is both conversational, "but also programmatic with prompts embedded in code," fellow ZDNET contributor David Gerwitz explains.
Also: Six skills you need to become an AI prompt engineer
Professional AI prompt-engineering job rates are hitting $175,000, but can be well over $300,000 per year, he notes, adding that "being a good AI prompt engineer involves more than being able to ask leading questions. You need to combine the disciplines of AI, programming, language, problem-solving, and even art to thrive on this career path."
With the world abuzz about generative AI, prompt engineers are in demand -- huge demand. The problem is that finding prompt-engineering skills is an intractable challenge. Recruiting prompt engineers is not for the faint-hearted. "I think that most people that are recruiting are stealing," quips Greg Beltzer, head of technology at RBC Wealth Management.
Also: Uh oh, now AI is better than you at prompt engineering
I had a chance to sit down with Beltzer at the recent Salesforce conference in New York, where he talked about the challenges of skilling up in the age of AI. "A good prompt engineer is more expensive than a data scientist today," he observes. "Just outrageously difficult trying to find somebody who has experience. You're not going to find someone who has more than five years of experience. At the most you might get two or three years, but it's hard to find."
Beltzer continues: "There is a dramatic need to get people up to speed on prompt engineering. But is it a science? Is it an art? Are we going to build more tools?" The good news is that once tooling is in place, it may be easier to train AI models with prompts conducted "systematically and programmatically", he says.
Yet until robust and useful tools are available, prompt engineering will remain a challenge. Even with tools, Beltzer says it's important to note that this skillset goes beyond technical acumen. What's more, it's too early to identify exactly the requirements and background that are best suited for a prompt engineer.
Also: Want a job in AI? These are the skills you need
For instance, Beltzer doesn't think it would make sense to train a data scientist or another adjacent professional to adopt prompt-engineering skills: "A lot of it needs to be business contextual. You need to think like the user to help with that prompt engineering -- it's not just code. It's not just development. It's like a business technical skillset that's also creative."
Some of the people coming into the field, he observes, aren't necessarily technical at all: "They're writers," he observes. "They just know how to write. And that's a part of it."
RBC keeps an eye on in its internal talent, with a focus on combined business and technical acumen, says Beltzer, "We're really looking for those folks that are most likely on the business side that has a technical bent. Personally, I don't want to make a bet until the tooling comes a little farther along."
The level of investment in AI and generative AI ventures during the past year, is "also going to shape what type of talent we're going to have," Beltzer says. "Until then, the talent market is going to be very lean. If you look at the turnover within these hot companies, they can name their price."
At RBC, which was once a highly conservative company, change has become the rule -- starting with its rising adoption of cloud-based capabilities and services, such as Salesforce. "Once we moved to the cloud, we've been doing 25 releases a year," says Beltzer. "Which for financial services is crazy -- the industry average was one release a year. We have a great team that is business and IT joined together, and we can iterate on the platform very, very quickly."
Also: How to write better ChatGPT prompts (and this applies to most other text-based AIs, too)
At the same time, Beltzer does not see his organization going all-in on AI anytime soon. While AI may help developers and business strategists with 80% of their tasks, the remaining 20% requires human involvement, he says: "I think AI is real. But I think we still have some work to do for the commercial viability in my industry."
For example, RBC employs generative AI to assist contact center engagements. "We have some pretty good use cases -- but it's cost mitigation, versus actual revenue generating," says Beltzer. "But it's a good start."
At a more general level, AI might never fully replace humans in the wealth management sector, he adds. "What we've seen in down markets is people don't want to talk to a machine telling them, 'It's going to be okay.' People want to hear, 'We built a portfolio that had this model for this type of scenario. You're still on track -- you're not retiring for another 20 years, you have plenty of time in the market, you continue to invest, it's going to be okay.' But you can't just have a bot telling you that."
AI capabilities are useful, however, in assisting employees as they talk directly to customers. "More and more people are using wealth management than ever before, because we have more assets," says Beltzer. "So, they're going to be able to service their clients with more technology -- making sure this box is checked, or that paperwork is done for them. That's where we need to scale. So, advisors can focus on the relationship with the client, and make sure what they invested in is going to meet their long-term goals."
As an IT manager, "our challenge is to make systems more scalable and more efficient," Beltzer says. "I need to make the human able to do what they love more and take away those baseline activities that don't add more value."