Human in the loop: Machine learning and AI for the people

HITL is a mix and match approach that may help make ML both more efficient and approachable.
Written by George Anadiotis, Contributor

Paco Nathan is a unicorn. It's a cliche, but gets the point across for someone who is equally versed in discussing AI with White House officials and Microsoft product managers, working on big data pipelines and organizing and part-taking in conferences such as Strata in his role as Director, Learning Group with O'Reilly Media.

Nathan has a mix of diverse background, hands-on involvement and broad vision that enables him to engage in all of those, having been active in AI, Data Science and Software Engineering for decades. The trigger for our discussion was his Human in the Loop (HITL) framework for machine learning (ML), presented in Strata EU.

Human in the loop

HITL is a mix and match approach that may help make ML both more efficient and approchable. Nathan calls HITL a design pattern, and it combines technical approaches as well as management aspects.

HITL combines two common ML variants, supervised and unsupervised learning. In supervised learning, curated (labeled) datasets are used by ML experts to train algorithms by adjusting parameters, in order to make accurate predictions for incoming data. In unsupervised learning, the idea is that running lots of data through an algorithm will reveal some sort of structure.

The less common ML variant that HITL builds on is called semi-supervised, and an important special case of that is known as "active learning." The idea is to take an ensemble of ML models, and let them "vote" on how to label each case of input data. When the models agree, their consensus gets used, typically as an automated approach.

When the models disagree or lack confidence, decision is delegated to human experts who handle the difficult edge cases. Choices made by experts are fed back to the system to iterate on training the ML models.

Nathan says active learning works well when you have have lots of inexpensive, unlabeled data -- an abundance of data, where the cost of labeling itself is a major expense. This is a very common scenario for most organizations outside of the Big Tech circle, which is what makes it interesting.

But technology alone is not enough. What could be a realistic way to bring ML, AI, and automation to mid-market businesses?

AI for the people

In Nathan's experience, most executives are struggling to grasp what the technology could do for them and identify suitable use cases. Especially for mid-market businesses, AI may seem like a far cry. But Nathan thinks they should start as soon as possible, and not look to outsource, for a number of reasons:

We are at a point where competition is heating up, and AI is key. Companies are happy to share code, but not data. The competition is going to be about data, who has the best data to use. If you're still struggling to move data from one silo to another, it means you're behind at least 2 or 3 years.

Better allocate resources now, because in 5 years there will already be the haves and have nots. The way most mid-market businesses get on board is by seeing, and sharing experiences with, early adopters in their industry. This gets them going, and they build confidence.

Getting your data management right is table stakes - you can't talk about AI without this. Some people think they can just leapfrog to AI. I don't think there will be a SaaS model for AI that does much beyond trivialize consumer use cases. "Alexa, book me a flight" is easy, but what about "Alexa, I want to learn about Kubernetes"? It will fall apart.

Not everything will be subscription based, and you have to consider that market leaders will not share the viable parts of their business. They may say so, but they don't -- there are many examples.


Gartner's analytics maturity model may be a good starting point to explain and prepare the transition to AI. Image: Gartner

Nathan says Big Tech tends to automate use cases that are easy for them, which means they do not involve interaction with humans, because that's not in their DNA. This is why he recommends that businesses work with second or third tier experts who are also aware of the business pain points.

And even if "outsourcing AI" was an option, it would still not be a great idea, for a number of reasons. Not just because of the bias in the datasets or the ethical reasons, but also because it is important to leverage knowledge within the organization:

"To get to artificial intelligence, you have to start with human intelligence. Knowledge does not come from IT, it comes from line of business and sales, and nobody outsources those."

Leveraging uncertainty

Nathan says he has witnessed a similar situation in the early days of Data Science, and although he acknowledges there is some distance to cover, he believes executives will eventually get it:

Decision makers are used to making judgments. Any CEO understands statistics at a gut level, because that's what they do every day. They may not know the math behind it, but the idea of collecting evidence, iterating on it and basing decisions on this is intuitive for executives.

Nathan says he believes in HITL because it's based on two things.

First, it combines teams of machines and people. Currently, very few people in HR have even considered this, but Nathan thinks that going forward we'll need things such as role descriptions for automation:

Many things we considered exclusively for people so far, will have to be considered for people and machines. If only for the sake of compliance, auditing, and continuity, these things have to be be taken very seriously.

In my experience all organizations have projects they do not embark upon because they do not have the resources. Once their processes are automated, people will be freed to embark on those.


Leveraging human expertise alongside AI is a way to develop and use AI.

Which brings us to the second point about HITL. Nathan points out that we typically think of ML as a way to identify patterns and generalize using lots of data, but we may be about to witness a paradigm shift:

Rather than just recognize patterns, we can use ML to look at data and identify opportunities, by identifying uncertainty. This is not about risk -- there's no upside to risk, you just buy insurance. But if you can separate risk from uncertainty, you can profit, because that's where the opportunities are.

With active learning, we have a way to identify where the uncertainties lie in our dataset. We can filter out the risk, and bring in the human experts to focus on the opportunity. We are starting to see companies that do just that, for example Stitch Fix.

I believe in this mixed model of augmenting experts. In any vertical, there is this proficiency threshold. You can achieve up to 80 percent proficiency, like someone who is good in that field. You can go up to say 95 percent, which is what experts achieve.

Beyond that, you get diminishing returns -- chaos and churn, judgment calls and experts disagreeing with each other. These are areas of exploration, there are no perfectly right answers, but we can actually leverage that.

Editorial standards