A guide for prescriptive analytics: The art and science of choosing and applying the right techniques
The first rule of prescriptive analytics is that you do not talk about prescriptive analytics—not before you've paid your dues in descriptive, diagnostic, and predictive analytics. That's not to say prescriptive analytics is not real, or does not have benefits. It does. But getting there won't come at the push of a button.
To see why, and what you need to do, we start by revisiting what prescriptive analytics is and go through a journey in the realm of analytics with a little help from the Gartners and Forresters of this world.
Prescriptive analytics is the final stage in the analytics evolutionary path
Analytics is the use of data, and techniques to analyze data, to get better insights and eventually make better decisions. Analyst firm Gartner introduced an analytics maturity model to reflect the fact that not all analytics techniques are born equal, and there is a progression in what you can achieve.
There is a chain of evolution in analytics, ranging from descriptive to diagnostic to predictive, and culminating with prescriptive, according to Gartner's classification.
Gartner's analytics maturity model may be a good starting point to explain and prepare for the transition to AI. Image: Gartner
Many organizations are still in the descriptive stage, utilizing more or less traditional business intelligence (BI) approaches: Get all your data together and use visualization to obtain quick views on what has happened.
Diagnostic analytics is about figuring out why an event happened and uses techniques such as drill-down, data discovery, data mining, and correlations. Most analytics frameworks have been incorporating such features in their offerings.
Where things get really interesting is when using predictive analytics to project what will happen. Typically this is done by using existing data to train predictive machine learning (ML) models.
Prescriptive analytics is the final stage in the analytics evolutionary path, with the ultimate goal being to provide ways of making certain outcomes happen.
In other words, predictive analytics tells us the likelihood of something happening, given current status on the basis of interpreting the data we have. Prescriptive analytics goes one step beyond and tells us what we need to do to make something happen. Prescriptive analytics draws a path on how to go from where we are to where we want to go. Let's pick an example to see this in practice.
A real-world example: Broken pipes and prescriptive analytics
A water utility company may start out its journey in analytics by deploying sensors in its distribution network and laying out the data infrastructure needed to feed that data to the appropriate databases--operational and analytical.
By making sure that data is sent to the operational database at all times, and replicated to the analytical database, the company will be able to see the status of its network in real time. This is descriptive analytics.
By connecting the analytical database to a software solution for analytics, and accumulating data over time, the company will be able to revisit data referring to incidents in its network. This way it may be able to figure, for example, that a broken pipe incident was due to increased consumption in the area. This is diagnostic analytics.
By accumulating data and analyzing incidents over time, patterns may begin to emerge. For example, the company subject matter experts and data analysts may be able to identify that when atmospheric pressure and temperature exceed certain thresholds, a broken pipe incident is likely to occur.
It takes a lot to make the most of what we usually take for granted: water.
Using historical operation data to feed a machine learning algorithm can help train the algorithms. Then, feeding the algorithm with current data, it can respond with a prediction on the likelihood of a broken pipe incident occurring. This is predictive analytics.
The next step would be to establish some sort of correlation between states in the data identified as problematic (leading to broken pipes) and states identified as normal. For example, when atmospheric pressure and temperature are above alarm-level thresholds, lowering the water flow or using alternative routes may help prevent a broken pipe incident.
By figuring out a mechanism that can be applied to state transitions, an analytics solution may be able to point out a path from a state that would lead to a suboptimal state to a desirable state. This is prescriptive analytics.
Prescriptive analytics techniques: The rules of optimization
So which techniques can help get from predictive to prescriptive analytics? Forrester Principal Analyst Mike Gualtieri, for example, mentions a number of technologies. But reading through those, it becomes evident that most of them actually apply to previous stages of analytics. Broadly speaking, the industry seems to converge around two sets of techniques: rules and optimization.
It has been noted that the 'Predictive Analytics Plus Rules' approach can be a simple way to get to prescriptive analytics. The rules-based approach combines predictions with business-defined rules and assumptions.
How would a rule-based approach apply to our water utility example? After having data that has been labeled as corresponding to a problematic state A, or to a normal state B, the challenge is to find the transitions leading from one to the other.
Domain experts, in this case field technicians or operations managers, may be able to identify the switches in the network required to go from state A to state B. These can be encoded as a set of rules, so when state A is identified in the data, the system will respond by suggesting to apply the corresponding rules to transition to state B. This works, but there are a number of caveats.
First, it requires extensive domain knowledge, and extensive involvement from the people who possess this knowledge. Second, the effectiveness of this approach is a function of the complexity of the domain. In the water utility example, if the network is sufficiently big and complex, it is likely that some state transitions can be missed or encoded in a suboptimal way.
Predictive analytics plus rules can lead to prescriptive analytics.
What experts are doing in this case is essentially searching a solution space to find transition rules, and then encoding these rules in a static way. The idea behind optimization is to search this space dynamically, in an automated way.
Optimization includes finding 'best available' values of some objective function given a defined domain (or input), including a variety of different types of objective functions and different types of domains. In other words, all states are evaluated via a function that correlates inputs to express utility. This approach may be more flexible and can also lead to better solutions.
In the water utility example, all inputs that influence states must be identified and encoded in a utility function. Then each state is evaluated and expressed as a numerical value via the utility function, and going from a state with low utility to one with high utility becomes a problem of search in the function's space.
Searching the optimization space to maximize a utility function may lead to solutions that are not obvious even for domain experts. AlphaGo approached playing Go as an optimization problem, leading to strategies that took even Go champions by surprise. The downside is that generating and testing a utility function that is realistic and efficient can be very hard.
Finally, machine learning may also be an alternative--as long as it is a branch of machine learning that can generate results approachable in a rule-based fashion. Those can be Decision trees, Fuzzy Rule-Based Systems, or Switching Neural Networks.
Conclusion
Prescriptive analytics is hard, and there's no silver bullet that can get you there without going through the evolutionary chain of analytics. You have to get the data collection and storage infrastructure right, the data modeling right, and the state classification and prediction right.
None of that is easy. But only then will you be ready to address the really tough problems that prescriptive analytics pose. The benefits, however, can be substantial. Prescriptive analytics is nothing short of automating your business.
Also See
- On the Rookout for live data: Instant observability to fix software bugs and open AI black boxes
- A standard for storing big data? Apache Spark creators release open-source Delta Lake
- Prescriptive analytics: A cheat sheet (TechRepublic)
- Prescriptive analytics: An insider's guide (free PDF) (TechRepublic)
- Feature comparison: Data analytics software and services (Tech Pro Research)