Microsoft and the ubiquity of data intelligence

Microsoft's got so much going on with data, analytics and AI, that at times its efforts seem chaotic. But these parallel efforts are coalescing, and in a way that harmonizes company strategy and lofty goals with research and the mainstream developer stack.
Written by Andrew Brust, Contributor

Microsoft's had a very busy couple of months. On September 29th, it announced, and I wrote about, several enhancements to HDInsight, its cloud-based Hadoop and Spark Big Data offering. Then, one day later, the company released the September update to Power BI.

Also read: Microsoft HDInsight gets Spark 2.0, faster Hive, and better security
Also read: Microsoft Power BI: A report card

Conference season
Along with the product releases, I had the opportunity last week to sit down and talk with Microsoft Corporate Vice President, Data Platform, Joseph Sirosh and his colleague Rimma Nehme, Technical Advisor, Data Platform. The conversation took place at in New York Strata + Hadoop World, where Sirosh had just keynoted, and did so right on top of another keynote at Microsoft's own Machine Learning & Data Science Summit event, also held last week, in Atlanta.

As if that were not enough fodder for a post, I've just returned from developer conference Visual Studio Live! in Washington, DC, where I co-presented a workshop on SQL Server on Monday and delivered a solo session on HDInsight and another on Power BI, on Thursday.

The conversation with Sirosh and Nehme provided me with Microsoft's take on all of its efforts. My talks at Visual Studio Live! gave me the opportunity to assess developer reaction to Microsoft's latest data and analytics updates, and to get very hands-on with both product updates so I could present them, properly. Put that all together, add in a day's pondering, and there are some good findings to share.

Powering through
First, consider that many of the changes in Power BI involve integration of that product with others. For example, one big update involves the preview availability of ESRI ArcGIS for Power BI. This product provides for the extended mapping technology that ESRI is known for as well as the ability to layer in a vast array of demographic data for the United States and assorted other data from around the world, including for the Nepal earthquake and New Zealand predator control.

Another change is an enhancement to the already present integration of the R programming language: now, instead of having to create and edit your R code inside Power BI and its rather spartan code editor, you can shell out to any R IDE (integrated development environment) and Power BI will copy both your code and your data there so you can work productively in a richer environment, before copy and pasting your edited code back into Power BI.

Also read: Microsoft's R Strategy

Further integration features included in the September update include new and enhanced connectors to various external data sources (all of them from other vendors, including Oracle and SAP); the ability to configure HTTP headers sent by Power BI's Web connector, which can be used to connect to RESTful Web service APIs from other services and vendors; and the addition of a new mobile view for reports and dashboards, a feature almost identical to the that of the Datazen product Microsoft acquired in April, 2015, and which is currently integrated into SQL Server Reporting Services.

R-n't we smart?
Speaking of integrations, Microsoft has also integrated the R programming language into SQL Server 2016, and has done so in a way that lets SQL Server act as a production deployment server for visualization-producing stored procedures and even for R predictive models. In fact, Sirosh mentioned to me that, with SQL Server R Services, Microsoft has been able to get SQL Server to support the generation of 100 million predictions per second.

Sirosh also mentioned that the company's Azure Data Lake Analytics service, which features a SQL-like language (called U-SQL) for querying big data, which is extensible through .NET code written in C# or non-.NET code written in R or Python. Add in things like Microsoft Cognitive Services and the company has now empowered Enterprise developers -- not just data scientists -- to utilize, embed and extend machine learning and artificial intelligence (AI) into their applications. In Sirosh's view, that takes us past Big Data and delivers us into the world of "Big Cognition."

Maybe that's why Microsoft recently created a division of the company focused exclusively on Research and AI.

Also read: Microsoft creates new combined AI, Research group

Russian literature, and intelligence
Rimma Nehme, meanwhile, showed me a little demo she whipped up on her laptop, using a combination of Microsoft technology, R and D3.js visualization code. To give you a sense of Nehme's big thinking, she showed me a network diagram visualization of all of the major characters in Leo Tolstoy's War and Peace, where each one could be clicked to reveal its visualized sentiment analysis, drawn from the (rather vast) text of the book.

Nehme, by the way, was previously a Principal Software Engineer in the Microsoft Gray Systems Lab, where she jump-started the PolyBase technology that connects SQL Server to Hadoop, as well as the cost-based query optimizer used in Microsoft Analytics Platform System (APS) and Azure SQL Data Warehouse. Nehme also holds a Computer Science PhD and an MBA, each from prestigious schools. She stands as an exemplar in proving that the absurdly male-dominated technology industry needn't -- and, above all, shouldn't -- be so. And despite her vast accomplishments, she's also incredibly pleasant and down-to-earth.

Also read: Microsoft's PolyBase mashes up SQL Server and Hadoop
Also read: Microsoft BUILDs its cloud Big Data story

What do the coders think?
Microsoft has traditionally been a developer-focused company, building platforms that developers can leverage in their own applications. The technologies mentioned here continue that tradition. And at Visual Studio Live! last week, they really seemed to capture attendees' attention and imagination.

At the conference's Day 2 general session, long-time figure in the Microsoft ecosystem, Tim Huckaby, showed off the integration of Microsoft Cognitive Services with Microsoft HoloLens, Kinect and even Huckaby's own iPhone. The software Huckaby showed, used in products and services delivered by the three companies he founded and leads (InterKnowlogy, Actus and VSBLTY), performs machine learning-driven facial recognition in support of security services in the companies' software.

Even with spotty WiFi coverage, Huckaby was able to add an audience volunteer to the database using his iPhone, and have that same person recognized and identified when he walked up to the Kinect. To say the Visual Studio Live! developer audience was fascinated would be an understatement. Their takeaway was that Microsoft was providing a platform that could make them data science/predictive analysis heroes in their respective organization. Reaction form the audience at my own Power BI session was similar and the sophistication of the questions was very impressive to me.

Intelligence! Developers! Intelligence! Developers!
Developers are starting to get that Microsoft is bringing machine learning and Big Data to them, instead of making them go into unfamiliar territory to learn how to work with Big Data and predictive modeling. And by bringing the technology to developers, Microsoft sees itself as bringing machine learning and intelligence everywhere.

Microsoft is the company that missed out on digital music players; never got enough apps on its smart phone platform; underestimated tablets; came late to, and has now retreated from, the wearables arena; and has weathered an ever-descending PC market. But it's a formidable contender in the cloud, and its data, analytics, machine learning and AI bona fides have depth, pervasiveness and a developer-friendly pedigree with the power to inspire, command loyalty and build lasting ecosystems.

Don't look now, but Redmond may well be in the lead.

Editorial standards