Machine learning may supercharge enterprise architecture

APIs as products in their own right, serverless architectures, and "legacy in a box" are the trends shaping IT management this year, according to ThoughtWorks' annual report on emerging technologies.

The report covered a host of techniques, languages, frameworks, tools, and platforms. I had the opportunity to get more perspective from Mike Mason, global head of technology at ThoughtWorks, who provided his take on where the enterprise development world is headed these days.

Machine learning is having a huge impact on enterprise sites, Mason says. "Machine learning is taking off because of a nexus of forces. First, the big data hype over the last few years means organizations are sitting on larger piles of data because they've been taught it might be valuable and they shouldn't throw it away. This data provides the foundation for useful machine learning." There has also been what Mason calls "a democratization of the algorithms. Google released TensorFlow as open-source, so now anyone can build a machine learning model, train it using the cloud or a GPU farm, and execute it at speed on a user's mobile phone."

However, Mason continues, don't expect to see machine learning helping developers in their own work anytime soon. "There have been many attempts at higher-order languages, where human developers simply explain their intent, and machine learning or AI conjures up implementation code. Practical usage of these techniques are a long way off."

At the same time, Mason points to "other development problems that are difficult but may be improved through machine learning." For example, he illustrates, "wrestling an existing suite of enterprise systems -- we have highly complex interactions between systems and components, complex data access patterns and relationships, and so on. Instrumenting these systems and then applying machine learning might yield insight that could allow us to do enterprise architecture better -- 'this system is a hot spot,' 'this system doesn't seem to be used much,' 'these two systems always behave in a related fashion, maybe they should be one system,' and so on."

A somewhat surprising absence from the ThoughtWorks report is the word "cloud," which is only mentioned a few times, though it's seen as the computing method everyone is adopting going forward. That's because it's now everywhere and everything, Mason points out. "Cloud is the defacto standard for computing, whether the underlying fabric is public -- Amazon, Microsoft, IBM or others -- or private -- Pivotal Cloud Foundry, RedHat's OpenShift or a custom private cloud," he says.

Cloud may be everywhere and everything, but don't expect any pullback in in-house coding anytime soon, Mason adds, "There has always been a build-versus-buy discussion and that discussion continues, although the buy portion could also mean using a service from a third party," he explains. "There's a whole slew of machine learning and AI [artificial intelligence] products available via API, for example, and organizations could choose to integrate one of those instead of attempting to build that functionality in-house."

In some cases, he adds, "an in-house installed product would perform worse -- these data-enabled services work best when they can operate on large data stores from more than one source, so a Google-provided machine language service might give better results than an in-house version."

The most important benefit, however, is "more things being available in the cloud or as services and APIs means that developers can build more ambitious solutions because they're standing on the shoulders of giants."

Mason also spoke to the rise of microservices and containers, which have "had a massive impact on enterprise systems." He reports that "almost universally, our clients want to adopt microservices, and the interesting thing is that there are organizational as well as technical implications. A trend we're seeing is that top technology execs are coming out of Silicon Valley and into more traditional organizations and then rebuilding their IT into the 'Amazon model.' That's the 'two-pizza team' model of small independent teams who are tasked with -- and empowered to -- drive a single business KPI."

What's important about the two-pizza team model "isn't the size of teams but the ownership they have over the KPI," Mason says, noting it invokes a "you-build-it, you-run-it ownership they have over their software, and the enlightened ways in which they interact with other teams -- things like API-as-product and treating other parts of the organization as though they are customers, even if they're internal. Microservices plus containers plus self-service platforms are basically required to implement these behaviors and interaction styles, so we're seeing a lot of traction there."

The latest ThoughtWorks report identified the following technologies and solutions as worth either adopting or strongly considering (or trialing) in the year ahead:

TECHNIQUES

Adopt:

• Pipelines as code

Trial:

• APIs as a product
• Decoupling secret management from source code NEW
• Hosting PII data in the EU
• Legacy in a box NEW
• Lightweight Architecture Decision Records
• Progressive Web Applications NEW
• Prototyping with InVision and Sketch NEW
• Serverless architecture

PLATFORMS

Adopt:

• HSTS
• Linux Security Modules

Trial:

• Apache Mesos
• Auth0
• AWS Device Farm NEW
• AWS Lambda
• OpenTracing NEW
• Unity beyond gaming

TOOLS

Adopt:

• fastlane
• Grafana

Trial:

• Airflow NEW
• Cake and Fake NEW
• Galen
• HashiCorp Vault
• Pa11y
• Scikit-learn
• Serverless Framework NEW
• Talisman
• Terraform

LANGUAGES & FRAMEWORKS

Adopt:

• Ember.js
• Python 3
• ReactiveX
• Redux

Trial:

• Avro NEW
• Elixir
• Enzyme
• Hangfire NEW
• Nightwatch NEW
• Phoenix
• Quick and Nimble
• Vue.js

VIDEO: Should carriers use machine learning to detect cellular network fraud?