The sustained hype around machine learning (ML) applications in the business world has some reasonable roots.
For one, it’s probably the most accessible and pervasive field of artificial intelligence (AI ) today. (AI and machine learning are closely related but not quite interchangeable terms.) ML is already embedded in many business applications, as well as customer-facing services. Also, it just kind of sounds cool, right? A machine, learning stuff.
[ Do you understand the main types of AI? Read also: 5 artificial intelligence (AI) types, defined. ]
Machine learning lessons: How organizations go wrong
As many an IT leader can tell you, though, excitement about a technology can lead to some unfulfilled and downright unrealistic expectations. So we asked a variety of ML and data science experts to share with us some of the tough truths that companies and teams commonly learn when they charge into production. Here are some important lessons from organizations that have learned them the hard way.
1. We didn't build the right team
You can have copious amounts of data and enough computing power to run every company in the Fortune 500, but it won’t matter if you don’t have the right people on your team.
“One thing that’s often under-emphasized is the tight-knit interdisciplinary team needed for a company to build its first few machine learning products,” says Jenn Gamble, Ph.D., data science practice lead at Very. “A data scientist rarely does this by themselves.”
ML success typically requires abilities that are virtually impossible to find in a single person or role. Gamble notes the following skills are usually key:
- Machine learning modeling
- Data pipeline development
- Back-end/API development
- Front-end development
- User interface (UI) and user experience (UX)
- Product management
“No one person is skilled in all these areas, so it’s essential that people with the right mix of skills are brought together and are encouraged to work closely throughout the process,” Gamble says.
[ Get our quick-scan primer on 10 key artificial intelligence terms for IT and business leaders: Cheat sheet: AI glossary. ]
2. We didn't build a bridge between business expectations and technical realities
Gamble also advises that the team responsible for any ML initiatives include people capable of working closely with subject matter experts and end-users elsewhere in the organization. Some (if not most) of them won’t be technical folks who understand the under-the-hood stuff.
“It’s important to have someone who is filling the role of AI Product Manager, even if that’s not their official title,” Gamble says. “Like a traditional product manager, their job is to be heavily focused on how the final machine learning ‘product’ will be used: who the end users are, what their workflows will be, and what decisions they’ll be making with the information provided.”
[ Learn how to define ML in terms everyone can grasp: How to explain machine learning in plain English. ]
Most IT pros can empathize with this issue no matter their particular skill set: There can be a bit of a gap (or a massive chasm) between the business expectations of what ML can do and the practical realities of its implementation.
“There’s also the added complexity of bringing together the business understanding, data understanding, and what’s possible from a machine-learning modeling perspective,” Gamble says. “In the same way that many of the best product managers are former software engineers, I suspect many of the best AI product managers will be former data scientists, although many other paths are also possible. The field is still so young there aren’t many people who have taken that path yet, but we’re going to see the need for this role continue to grow.”
3. We had too many versions of the truth
One basic reality of machine learning: A model or algorithm is only as good as the data it feeds upon.
“The key thing to remember about AI and ML is that it’s best described as a very intelligent parrot,” says Tom Wilde, CEO at Indico. “It is very sensitive to the training inputs provided for it to learn the intended task.”
But this leads to a different kind of learning: There can be significant variability in how people – even those on the same team – perceive the reality of a particular business process or service.
Wilde’s firm enables a customer to have multiple people participate in the process of labeling training data for the purposes of building a model. He thinks of it like voting: Each stakeholder gets a say in the process or task. Recently, a client had a half-dozen people participate in the data labeling process, which led to a short-term failure but a longer-term gain.
“Once the model had been built, they discovered that the model’s performance was extremely poor, and upon further investigation, they found that the six individuals had quite different views on how to label the training samples,” Wilde says. “This in turn forced a very valuable conversation about the particular task and enabled them to better codify the ‘ground truth’ understanding of this particular use case.”
4. We thought our training data was a finish line
You also might find out in production that you were a little too confident in your initial training data in a different manner, by shifting to the past tense: trained. Even great training data isn’t necessarily enough, according to Jim Blomo, head of engineering at SigOpt.
“You can’t just train a model and believe it will perform,” Blomo says. “You’ll need to run a highly iterative, scientific process to get it right, and even at that point, you may see high variability in production.”
The same holds true of your simulation and validation processes, as well as ongoing performance measurement.
“Teams will often find that the benchmark used to project in-production model performance is actually something that needs to be adjusted and tuned in the model development process itself,” Blomo says. “One of the first things modelers typically learn is that defining the right metric is one of the most important tasks, and typically, tracking multiple metrics is critical to understanding a more complete view of model behavior.”
5. We repeated traditional software development mistakes
Machine learning is susceptible to the same issues that can plague the rest of IT. Did you build your AI/ML team in functional silos that don’t work cohesively together? That’s going to cause many of the same problems that hampered traditional software projects: Think scope creep, blown deadlines, broken tooling, and festering culture woes.
“Companies spent years collecting big data, hired teams of data scientists, and despite all that investment, failed to get any models in production,” says Kenny Daniel, founder of Algorithmia. “The wrong answers are to have data scientists throw code over the wall to an implementation team, but it’s also wrong to expect data scientists to be DevOps experts.”
The right answer? Apply the same kind of thinking (such as DevOps thinking) you’ve used to modernize and optimize your software pipeline to machine learning.
[ Read also: 3 DevOps skills IT leaders need for the next normal. ]
“Learn the lessons of DevOps in the traditional software world: Create automated, repeatable pipelines and tooling to containerize and abstract away the underlying implementation details,” Daniel advises.
This is a team-building issue, too, and it intersects with Gamble’s reality check above.
“All of the same principles and lessons learned from software development – DevOps principles, user-centered design, et cetera – are still necessary when building machine-learning products,” Gamble says. “Many data scientists have spent a lot of time learning about machine learning and are valuable for that reason, [but they] might not be as well-versed in these topics as software engineers, product managers, or designers are.”
Just as DevOps can be seen as a widespread response to stubborn problems in traditional software development, so too are new fields and approaches already emerging in machine learning and other spokes of the AI umbrella.
“Because of the additional considerations required when incorporating machine learning into the traditional product development mix, new fields are blossoming, like MLops, DataOps, DataViz, and MLUX (machine learning user experience), to try and fill this gap,” Gamble says.
[ How can automation free up more staff time for innovation? Get the free eBook: Managing IT with Automation. ]
