How big data and AI work together

How is artificial intelligence – and its prominent discipline, machine learning – helping deliver better business insights from big data? Let’s examine some ways – and peek at what’s next for AI and big data analysis
729 readers like this
Crossing the gap to big data

4. AI/ML can be used to alleviate common data problems

Here’s something that hasn’t changed: The value of your data is inextricably linked to its quality. Poor quality means low (or no) value. This is something that so-called big data has in common with AI.

“The ‘dirty’ secret of ML projects is that 80 percent of the time is spent cleansing and preparing the data.”

“Every conversation about machine learning always comes back to the quality of the company’s data. If the data is dirty, any insights derived from it cannot be trusted,” says Moshe Kranc, CTO at Ness Digital Engineering. “The ‘dirty’ secret of ML projects is that 80 percent of the time is spent cleansing and preparing the data.”

Everything old is new again, it would seem. But the solution to this problem (and potentially others like it) might already be staring you in the face.

“Fortunately, machine learning data can be cleansed using… machine learning!” Kranc says. “ML algorithms can detect outlier values and missing values, find duplicate records that describe the same entity with slightly different terminology, normalize data to a common terminology, etc.”

5. Analytics become more predictive and prescriptive

An ML algorithm can be taught to make a decision or take an action based on a forward-looking insight.

In the past, data analytics was more postmortem than anything else: “Here’s what happened.” Future predictions were still essentially historical analyses. AI and ML are helping open a new front: “Here’s what’s going to happen.” (Or at least “here’s what likely going to happen.”) Moreover, an ML algorithm can also be taught to make a decision or take an action based on that forward-looking insight.

“Today, AI is moving big data decisions to points further down the timeline, in more accurate ways, by using predictive analytics,” says Sean Werick, managing director of analytics at Sparkhound. “Traditionally, big data decisions were based on past and present data points, generally resulting in linear ROI. With AI, this has grown to epic and exponential proportions. Prescriptive analytics, leveraging AI, has the potential to provide company-wide, forward-looking strategic insights helping to advance the business.”

Werick notes that this is a “learn to crawl before you walk” progression. Using AI to make predictive or prescriptive business decisions based on inaccurate or inadequate data could have “catastrophic” outcomes, according to Werick. But this is the progression AI is enabling.

“The value to the business increases with each progression through the analytics maturity model: beginning with process and data mapping, to descriptive analytics, to predictive analytics, and finally, to prescriptive analytics,” Werick says.

6. What’s next for AI and big data? We’ve merely scratched the surface

If most teams are still learning to crawl (or walk), that might be OK because the combination of AI and big data is just beginning to reveal its possibilities.

Andy Vitus, a partner at Scale Venture Partners, sees a big future in more intelligent enterprise software, for example. Many business applications still show their analog DNA, in his view.

“Users are still spending inordinate amounts of time slogging through endless reports.”

“Most business apps are still built using the design language of paper forms and ledgers. This means that for all of the data being captured and stored by enterprises, users are still spending inordinate amounts of time slogging through endless reports to find useful information,” Vitus says.

“The future is intelligent software that leverages all of that data to solve problems and do work for us – providing context and answers rather than just nicer-looking reports. From an engineering perspective, intelligent enterprise applications will require that we connect individual AI/ML systems to other systems so that they can communicate with and learn from each other. Enterprises will finally see significant ROI from all of that data they’ve been storing.”

That’s the essential promise: AI as an evolving means of answering that basic question (see #1) about big data: Now what?

“This is just the beginning – in the future, there will be new techniques that emerge on how to analyze data for real-time insights,” says Mih from Alluxio. “The data is still the data, but the ways of getting insights on it will improve.”

[ How can automation free up more staff time for innovation? Get the free eBook: Managing IT with Automation. ] 

Kevin Casey writes about technology and business for a variety of publications. He won an Azbee Award, given by the American Society of Business Publication Editors, for his story, "Are You Too Old For IT?" He's a former community choice honoree in the Small Business Influencer Awards.

Social Media Share Icons