Big Data and analytics are powerful, and the technologies around them are exciting. But if they can only be harnessed by highly-paid specialists, then they haven't fully evolved. We need data and analytics technologies, but we shouldn't need expensive, scarce, Shaman practitioners to use them. More than data scientists, we need tools that empower knowledge workers to do Big Data analytics on their own.

Is crossing over possible?
People can certainly obtain the literacy necessary to carry out analytics on Big Data.  Business people can be made capable of working with the data, and developers who are not currently analytics-focused can be made capable of collecting the data and performing analytics in their code.

Of course, certain people can be trained to become very highly-skilled specialists, but that would be the exception more than the rule, and that's OK.  We don't need people to retool en masse into scientists, we need them to obtain a competency.

Beyond the hype
The term "data scientist" is over-hyped.  But in fairness, so are the terms "Big Data" and "analytics," and yet these are still quite valid areas of specialization.  The problem with the term "data scientist" goes beyond the hype; there's an attitude and adversarial tone to the term. This tone discourages people from obtaining analytics competency, as it transmits an implicit message that the work must be outsourced to highly-trained individuals.  Aside from the hype, it's pretension and snobbery that make "data scientist" an unhelpful term.

Dilution of the term
There a risk that many technologists will become "data scientists" in the name of finding a better gig, in exactly the same way that happened with other lofty titles in technology ("architect," for example).  Title inflation happens in any field, but in the tech field, terms and titles are in any case viewed as metaphors, more than literal descriptions.  Tech folks tend to take poetic license with titles, and those who don't do so find themselves at a disadvantage compared to those who do.

It's the tooling, stupid
Analytics in general, and Big Data specifically, have terribly immature tooling compared to mainstream relational database and BI products.  That being the case, it's no wonder that only "scientists" can get real work done.  These tools were built for laboratory use, not business use.

Just as self-service BI is in vogue (and is legitimately quite powerful) today, so too should self-service Big Data and predictive analytics be a market phenomenon.  Once it is, people with the skills that we classify under data science today will still have an important role, but it won't be nearly so central as it is now.

Data literacy, and what it could look like
It won't come as a surprise that I believe a scenario where we have more data literacy -- and business and tech people who are "bilingual" -- to be the one that will most successfully solve the labor issues we face.  Data science is about having a command of both data technology and business domain expertise.  If the technology becomes simple, and business people become more adept with it, then business users can be bona fide analytics professionals.

If I had my druthers, a perfect business analytics wonk would be a sales, marketing or planning professional who was also a tech power user, had a command of statistics, knew Excel very well and could do some light programming.  But that's an ideal…and in order for analytics technology to take off, we shouldn't need people to fit this ideal in order to be productive Big Data analysts.

Data science algorithms?
Implicit in the definition of data scientist is possession of business intuition and instinct that mere algorithms can't replace.  If you accept that the term is legitimate, then you accept that a combination of human intelligence and technology expertise is what makes someone an authentic data scientist.  While I'm not a huge fan of the term "data scientist," I do feel the experience of the business user and her non-algorithmic intimacy with the semantics of the data is very important.

What we will need, and what we won't
Expertise in data exploration and visualization tools, programming/developer skills, an understanding of statistics, and high-level database design skills will remain important, regardless of whether the data scientist role remains in vogue.  Equally important will be a deep understanding of the business, and the data sources that measure its activity and outcomes.

The term "data scientist" will subside and may well sound dated five years from now.  The skills will become more commonplace and commoditized.  When that happens, the real boom will begin, because the technology will become widely adopted and thus more useful.  But for the relatively small club of people clinging to a data scientist identity and pay scale, it may seem like a bust.

Summing up
Big Data technology is powerful, and it keeps getting better. But the technology does, right now, require niche specialists to derive the greatest business value from it. These specialists have to be renaissance people – possessing a combination of technology, mathematics and business skills, and knowledge. It's not clear that being so clever and versatile makes these specialists into "scientists," but it does make them rarefied.

Nonetheless, for Big Data and analytics implementations to grow and become truly mainstream, having such diverse skill set requirements for them is not a sustainable situation. Market need is going to drive evolution in the technology such that the barrier to entry will not be nearly so high as it is now. If for some reason that didn't happen, then adept use of Big Data would continue to be an option open only to a relatively small group of customers.

The solution to our problem isn't legions of new data scientists. Instead, we need self-service tools that empower smart and tenacious business people to perform Big Data analysis themselves. The specialists will still have an important role, but they won't be the linchpin to scaling Big Data across industries.

SiSense, a company on which I have reported before, has introduced a new version of its Prism product, dubbed Prism 10X, which delivers major enhancements to its underlying ElastiCube database engine.  The company says the new version provides 100 times the data capacity and 10 times the speed as competing, in-memory analysis solutions running on on the same hardware.

Think global, act local
A lot of available Big Data solutions work on a "scale-out" approach to processing data. Typically this means adding more commodity servers to a cluster, so that more data can be processed in parallel, allowing processing times to remain reasonable even as data volumes increase.

Such an architecture is very powerful, but it can take focus away from the parallel optimizations that can be had on a single machine.  These in-machine optimizations are exactly where SiSense's engineers have concentrated their efforts, employing a combination of cache-awareness, columnar compression, predictive pre-fetching and vectorization.  

Cache flow
SiSense's ElastiCube engine focuses not just processing data in-memory, but also within a Central Processing Unit's (CPU's) on-board cache.  Moving data in and out of cache is much faster than doing so with Random Access Memory (RAM), and while many data engines use cache only incidentally, Prism targets in-cache data manipulation explicitly.

Cache is much smaller than memory, so the ElastiCube engine employs columnar compression, not just for the storage of data on disk, but also for its in-cache persistence.  The engine also factors out queries into sub-queries (which SiSense calls "instructions" and says tend to repeat) and pre-fetches results for the sub-queries that its heuristics tells it users will want.  Interestingly, these heuristics improve as the engine's workload increases, so greater load on the system can actually lead to better performance.

Prism not only targets cache, but makes use of newer CPUs' "single instruction, multiple data" (SIMD) instructions, which process several data values at once, rather than one at a time.  This facilitates parallel processing within a machine, rather than between nodes (servers) in a cluster.  This technique is sometimes referred to as vectorization.

Start your engine...and then keep going
As obsessive as the SiSense engineering team is about crafting a super-efficient query kernel, Prism is more a competitor to data discovery and visualization tools like Tableau, QlikView and TIBCO Spotfire than it is to a data warehouse or online analytical processing (OLAP) products.  Prism includes scatter charts, wind-roses, funnels, scatter and areas maps, among others, and SiSense says that "thousands of combinations are available."

Competing data discovery tools have their own engines too, and use a combination of columnar and in-memory techniques to attain high performance.  But they don't seem to exploit cache and SIMD operations (what SiSense calls "in-chip analytics").

Room to grow
SiSense is far from perfect.  As good as its single node optimizations are, its lack of cluster-based deployment capability will be a turn-off to some who are looking for petabyte-scale solutions.  But for data discovery work, terabyte-scale is where many (if not most) enterprise customers are right now.

Clustering capabilities may come in a future release.  But for now, SiSense is focusing on a high-speed, integrated solution for single-box data discovery work, and its reported 520% year-on-year growth is making the company feel its approach is quite well-validated.

Today was a big day for Big Data and analytics, as data discovery and visualization darling Tableau Software made good on its filing for an initial public offering.  The company's shares were issued today on the New York Stock Exchange, under the ticker symbol "DATA."  Wow.

Tableau offered 8,200,000 shares of its Class A common stock at a price to the public of $31.00 per share.  The shares closed up almost 64% above that initial pricing today, their first day of trade.  Tableau now has a market capitalization of $2.9 billion.

I had the opportunity to speak today with Tableau's CEO and co-founder, Christian Chabot, and I discussed with him matters relating to Hadoop, data scientists, the business intelligence space, its business models, and what's next for Tableau.

Don't spend it all in one place
To start, I asked Chabot what the IPO-derived capital would be used for.  His answer was that the IPO was more about raising public awareness of Tableau, and the credibility of the company, than it was for expansion per se.

"Tableau everywhere" is what Chabot says is the company's next frontier.  He explained that Tableau's revenue is currently derived from a Windows-only, on-premises-only offering and one with limited market awareness, to boot.  So there's a lot of growth opportunity to go.

Product or stack?
While that's all well and good, we have to assume that Tableau will expand its sales force and quite possibly its product portfolio.  So I asked a few questions around those topics.

With Tableau's growth as a private company, and its now seemingly quite successful IPO, it has set a very high bar for itself.  I pointed out to Chabot that this has all been built around what is essentially a single product, and so I asked what Tableau would do to keep the momentum going.

First off, Chabot corrected my assertion that Tableau is a single product, insisting that Tableau Desktop, Tableau Server and Tableau Public are quite separate.  I suppose this comes down to semantics; to me, three different editions that are all geared to data discovery don't constitute separate products, but certainly they are marketed separately, and that does count for something.  

I peristed in exploring the possibility of new products from Tableau though, as most of its BI competitors offer a full stack of products that cover data integration, master data management, data quality, conventional reporting, and more.  Chabot explained that there's little reason to match everyone with a full BI stack simply for the sake of conforming to the market category.  But he also told me that the company is interested in diversifying into new product areas for which Tableau is seeing significant customer demand.  Chabot said that a data integration offering is of particular interest to Tableau.

Declaring independence, and neutrality
But if Tableau remains mostly focused on data discovery and visualization, it begs the question of whether it will be acquired by one of the BI stack vendors that is weak in that area (and compared to Tableau, many such vendors are).  Chabot insisted that Tableau will remain independent, explaining that such independence allows the product to remain a Swiss Army knife that connects to virtually any relational, Big Data or analytical data source, and how that benefits Tableau customers greatly.

Certainly, Tableau customers are a happy bunch, as I noted in my report from the Tableau Customer Conference last year.  Chabot believes strongly that Tableau's undiluted dedication to self-service is what drives customers' passion, and that it also put Tableau well ahead of its BI competitors that offer self-service capabilities as a mere option, if at all.

Data science, for the layman
Having just participated in ZDNet's Great Debate, on the need for data scientists this week, I asked Chabot what he thought about the issue.  Not surprisingly, Tableau's CEO feels that we are too reliant on specialists, and that expanding this "priesthood of people" doesn't get us past the bottleneck.  Chabot said that the complicated and developer-intensive nature of the vast majority of data technologies is what underlies Tableau's success.

With that complexity in mind, I asked Chabot about Hadoop.  The quintessential Big Data technology is clearly popular, but hardly something one thinks of when the self-service, agile and empowerment themes that Tableau identifies with are invoked.

"DATA" Convergence
How did Chabot reconcile the success of self-service with that of a complex tool like Hadoop?  He explained that Tableau sees Hadoop at many customers, but almost never sees it as a standalone platform.  Chabot's implication was, I think, that people want to use Hadoop, but they want it to meld with the data warehouse, BI and transactional database technologies they have been using for some time.  Chabot would tell you that Tableau facilitates some of that integration, and he'd be right.

I will just point out that 2013's trend of BI - Big Data convergence shows no sign of slowing.  Tableau's ticker symbol of "DATA" doesn't have "BIG" in it, and it doesn't need to, because the market need around data of all types is even bigger.

Now that Dell has acquired Quest, and made it part of its $1.5 billion software unit, the data acquisition and analysis tools that Quest brings to the table have strategic importance beyond their value as mere IT operations tools. Database replication is important for fault tolerance, branch operations, and more. But Dell has clearly decided that such an engine can and should also be used to move data between platforms, to aid in data integration and analysis.

Dell has effectively added an open output pipeline to SharePlex, implemented using a Java Message Service queue. It then created a connector that subscribes to the queue, and pushes the extracted data to Hadoop using Sqoop, which leverages Hadoop's MapReduce engine to move data in and out of Hadoop's Distributed File System(HDFS).

As the diagram below shows, developers can also build their own custom apps that subscribe to the JMS queue, and subsequently push the data into other platforms, or they can use existing data integration tools to do likewise:

If you're not up for the custom solution, though, fear not. Dell will be adding its own connectors. According to Dell Software's Darin Bartik (executive director of product management for information management solutions) and John Whittaker (director, marketing), Microsoft's SQL server is on the shortlist here. The logos in the diagram may or may not indicate other databases to be directly supported in the future.

Dell Software has a partnership with Cloudera, so we can expect that SharePlex's Hadoop connectivity will work very well with that company's Hadoop distro. However, Bartik and Whittaker assured me that Hortonworks' and MapR's distributions are supported, too.

With the release of SharePlex 8, Dell Software is taking big data convergence a step further than we've become used to seeing. Not only are the worlds of big data and BI colliding, but now, relational databases and OLTP are part of the equation as well.

One NewSQL database, NuoDB, also runs on both operating systems. But the folks at NuoDB looked at their downloads, and discovered that a full 80 percent of them were attributable to the Windows edition of the product. Rather than bury their heads in the Linux sand regarding this phenomenon, the NuoDB folks took ownership of it instead.

(.Net) Developers, developers, developers

The result is version 1.1 of the NuoDB Starlings database. This version ships with support for Microsoft developer technologies to a degree that is almost unprecedented for a third party. These creature comforts include an ADO.NET provider, a LINQ (Language Integrated Query) provider, and an Entity Framework provider that supports the code-first and model-first approaches.

NuoDB 1.1 also includes a migration assistant for moving databases off the SQL Server platform and onto NuoDB. This is a shot across Microsoft's bow, although from a relatively small vessel. Regardless, the value proposition is interesting. While SQL Server can be implemented in a clustered configuration, that's not easy to do, and its architecture is mostly geared toward single-node installations. Additional SQL Server nodes are often made available for fault tolerance purposes, more than for distributed processing.

How it works

NuoDB is a scale-out, cloud-oriented database, built to be geographically distributed by default. Its architecture consists of broker nodes, transaction nodes, and storage management nodes. Each transaction node is responsible for its own "atoms" — little pieces of the database, be it data rows or parts of indexes, that communicate asynchronously. The atoms use multi-version concurrency control (MVCC) to maintain database consistency and ACID guarantees.

The broker nodes handle load balancing between the transaction nodes, and as long as there are two or more broker nodes, NuoDB avoids having any single points of failure. Storage management nodes keep track of the atoms that make up a database, and their replicas.

Nodes in a NuoDB cluster act together on a peer-to-peer basis, analogously to birds in a flock, hence the "Starlings" name and the company's logo design. The whole scheme leads, if the company's claims are accurate, to a very low-latency, geo-distributed database that still uses the relational model, transactional guarantees, and SQL query language that are standard for enterprise developers.

NuoDB nodes can run in the cloud (NuoDB now explicitly claims Windows Azure compatibility) and, interestingly, NuoDB clusters can consist of a mixture of Windows and Linux nodes.

Should Redmond worry?

It's not at all clear to me that NuoDB will siphon off hordes of SQL Server databases and customers. Of course, I'm an SQL Server MVP and a co-author of a book on Microsoft's flagship database, so I have my biases. But coming from that background, NuoDB seems more tempting to me than MongoDB or Cassandra, for example.

Allowing .Net developers to move to a scale-out database architecture — and yet keep their existing APIs, database schemas, and query language — poses a formidable challenge to Redmond.

But HANA requires the entire database to be in memory, and while memory prices have declined dramatically, memory still commands a huge premium over disk, on a terabyte-by-terabyte basis. Plus, a single server can accommodate only so much memory in the first place, so petabyte-scale data work on HANA can require huge hardware expenditures.

Teradata memorizes

Teradata sees the value of the in-memory database movement, but the company likely can't ask its customers to take the gold-plated approach of using memory exclusively, with disk storage present only as a fail-safe. Further, the company has studied the probabilities, and found that after a certain percentage of your data is stored in memory instead of disk, the returns quickly diminish. In fact, Teradata found that 43 percent of all IO (input/output) hits only 1 percent of all disk cylinders, and that 94 percent of IO hits only 20 percent of all cylinders. Talk about your 80-20 rules!

With that in mind, the company is announcing the addition of "Intelligent Memory" functionality to its venerable database appliances. Intelligent Memory will see to it that the most frequently used (the "hottest") data in a database stays resident in a special extended memory region and infrequently queried ("cold") data stays on disk.  The determination of which data is hot and cold is updated dynamically, with Teradata consequently moving certain data in and out of RAM, at opportune times, in terms of processing inactivity on the cluster.

Combination of approaches

While this may at first sound like a caching scheme, that's not the case. A cache typically moves recent data into RAM, rather than frequently used data. That's a simpler approach, but it's also valuable. Teradata employs a caching approach as well; what it calls the FSG (File Segment) cache. And Teradata is smart enough to make sure that no data kept in the FSG cache will be moved into Intelligent Memory, and vice versa.

The 14.0 release of Teradata added columnar storage technology (whereby all the values for a column/field are stored contiguously, rather than all the values for a row/record/item being stored together). This allows for high rates of compression, as column values are often close in value, and certainly in order of magnitude. In the upcoming Teradata 14.10 release, Intelligent Memory will recognize columnar storage and maintain it. This means that high rates of compression will be maintained as well, allowing more data to fit into Intelligent Memory. It also means that if only certain columns' data is "hot", and that data is in columnar storage, then only those columns' data will get moved into Intelligent Memory, yielding further efficiencies.

You get in-memory, and you get in-memory, and you get in-memory

Interestingly, while the FSG cache is only available on certain Teradata editions, Intelligent Memory is available on all of them. So all customers will benefit, and since new, high-memory cluster nodes (or a cloud service) are not required, even smaller customers can look forward to better performance.

The combination of columnar compression, caching, and in-memory placement of hot data means that Teradata's approach to the in-memory craze is reasoned and has customers' interests in mind. I might prefer to think of the feature as "Reasonable Memory", but I guess that doesn't have quite the same ring to it.

Talend's DI platform

But Los Altos, California-based Talend, a specialist in Data Integration (a classier moniker than ETL, I suppose), offers an open source distribution of its platform. As a result, Talend has a vibrant ecosystem that often results in new community-developed connectors for the platform in addition to the more than 450 connectors included natively in the product.

And the Talend platform includes more than just plain ETL: Master Data Management (MDM), Data Quality, Business Process Integration (BPI), and even an Enterprise Service Bus (ESB) are part of the mix.

Get your Hadoop on

Talend version 5.3 now features a graphical mapper for building 

Talend 5.3 can also generate native Java MapReduce code, which allows data transformations to run right on the Hadoop cluster, avoiding burdensome data movement, and making use of general purpose SQL and import/export tools like Hive and Sqoop unnecessary.

NoSQL, no peace!

Talend 5.3 also adds to its NoSQL connectivity capabilities. While the prior release could connect to HBase, Cassandra, and MongoDB, v5.3 adds support for Couchbase, CouchDB, and Neo4j. This provides coverage for the  most popular NoSQL platforms (aside from proprietary offerings like Amazon Web Services' DynamoDB). It also means Talend has connectivity to databases across all four major NoSQL categories (key-value stores, document stores, wide column stores, and graph databases).

Will loyalties shift?

Whether Enterprises will go "cold turkey" from standalone DI tools, like those from Informatica, or sophisticated, bundled ETL tools, like Microsoft's SQL Server Integration Services, remains to be seen. But there's a lot to be said for graphical tools over Hadoop, native MapReduce code, connectivity across major NoSQL data stores, and the option to work with open source distributions of a product before standardizing on it.

Think search has nothing to do with big data? Think again, because Lucene and Hadoop have a special relationship. To start with, Doug Cutting, now chief architect at Cloudera, is the man behind both projects. Next, Lucene can work over HDFS; process data stored in HBase; and Mahout can use Lucene indexes. Lucene and Solr are also included with certain distributions of Hadoop. In fact, as I reported earlier this month, MapR's Hadoop Distributions include the full LucidWorks suite, which is based on Lucene/Solr (and LucidWorks, is the major commercial entity behind Lucene).

Today, Lucene/Solr 4.3.0 was released and made available for immediate download. The 4.3.0 package includes improvements in numerous areas, including query performance, spatial processing, and the read-side schema API. 4.3.0 also includes numerous enhancements to Lucene's faceted search capabilities, whereby dimensions can be used in search, much as they are used for drill-down analysis in data warehouses and OLAP cubes.

Full details of the release — and links to download both the Lucene and Solr components of it — are available at the Lucene project's home page. And if you're intrigued by all this, check out the website for Lucene/Solr Revolution, LucidWorks' first annual conference on Lucene and Solr, which was just held in San Diego.

MapR's M3 and M5 Hadoop distros are perhaps best known for providing a non-immutable network file system in place of the standard Hadoop Distributed File System (HDFS) implmentation.  MapR Hadoop is also known for its cloud availability, on Amazon Elastic MapReduce (EMR) and Google Compute Engine. In March, MapR announced that its Hadoop bits would be distributed with Ubuntu Linux.

This one goes to M7
Today MapR is announcing the general availability of its new M7 distribution, which includes a re-engineered version of the HBase wide column store NoSQL database.  Essentially, M7's HBase implementation makes it far more practical for operational workloads, due in part to MapR's underlying read/write file system as well as optimizations to the HBase code itself.  (I covered these enhancements, back in October when MapR announced the Beta release of M7.)

MapR is also announcing a partnership with LucidWorks, a company which says it employs one fourth of the core committers to the Apache Lucene/Solr project.  That project provides popular, powerful open source search capabilities across numerous technology platforms, including HDFS.  MapR will include the LucidWorks suite (which includes added functionality over the vanilla Lucene/Solr code) with its Hadoop distribution, making it possible to query Hadoop through plain-language search in addition to Hive's HiveQL and Pig's "Pig Latin" query languages and, of course, MapReduce code written in Java.

LucidWorks' integration with MapR's M3 and M5 distros is available now, as a Beta.  The company says that LucidWorks will be integrated with M7 sometime next quarter.

All data, all the time
1010data is a cloud-based data warehouse, data visualization and data discovery solution.  The product uses columnar database technology and a spreadsheet-like user interface to facilitate self-service analysis by business users over large volumes of data (the company calls it the "trillion row spreadsheet").

1010data, which has been in business for 13 years, and whose customers include the New York Stock Exchange, Rite Aid and Dollar General, is today releasing version 6 of its product.  The new version includes an enhanced user interface; a new version of 1010data's data-integration tool; improved in-database analytics; new machine learning functions; a "Quick App Builder;" and a new customer administration portal.

Big (Data) news cycle
I'm not exactly certain why there have been so many new product, partner and release announcements this week (and Teradata and Tibco just added another one).  But certainly, there seems a real urgency to make Big Data technology more capable, more accessible, and more integrated with other tools.  And there's nothing wrong with that.

I've spent most of my career in the Enterprise development and tools space, so I suppose I could be jaded about these announcements.  But having been an avid Big Data watcher for more than a year, I must admit I wasn't sure I'd see the day when SQL, Enterprise security, monitoring and seamless provisioning would make up the headline fodder in this space.

And yet, that's exactly what's happening today.  Let's take a look at the details...

Cloudera annouces general availability of Impala 1.0
Back in October, Cloudera shocked me a little by telling me they believed MapReduce wasn't the solution for all Big Data problems, and that they were building a parallel SQL query engine that could work over data in Hadoop's Distributed File System (HDFS) and bypass Hadoop's MapReduce engine, permitting speedy iterative (non-batch mode) query over Hadoop data.  Today that product, Impala, has reached GA.  And since it's not only fast, but API-compatible with Apache Hive, lots of existing BI tools can work with it right away.  

This fits nicely with Cloudera's announcement yesterday that it has formed an alliance with BI powerhouse SAS.  That alliance is not just a business arrangement either, as SAS engineers have adopted their technology to deploy physically over Hadoop clusters and perform their analyses in a parallel fashion.  This is a huge deal as it avoids data movement between SAS and Hadoop,  analyses can be performed over full data sets and not just samplings of the source data.

Splunk releases version 2.4 of its App for Enterprise Security
The combination of Splunk's App for Enterprise Security brings to bear statistical analysis of user-generated machine data to discover unknown threats to digital systems in real time.  Rather than merely monitoring for known threat patterns, Splunk's suite uses search, dashboards and visualizations to detect anomalies and outliers that may indicate the presence of yet-unidentified threat patterns instead.  Clearly, bringing hardcore statistical analysis to users, rather than just raw Big Data storage and query capability, is where real value gets added.

MongoDB gets a backup service
10gen, the company behind venerable NoSQL document store MongoDB, is today announcing the limited release  of MongoDB Backup Service, joining the free, cloud-based MongoDB Monitoring Service (MMS) it launched previously.  Given my own 20-year background in relational databases, the lack of a dedicated backup service seemed a bit strange to me for such a cloud-friendly product as Mongo.  But the Big Data and NoSQL worlds are no strangers to gaps like that.  The more important point is that the gaps are being filled very rapidly this year, and 10gen's announcement is part of that trend.

Basho and SoftLayer provide Riak in the cloud
Speaking of NoSQL and cloud friendly, Basho, the company behind NoSQL key-value store Riak, has teamed with cloud provider SoftLayer, to provide an easily provisioned option for running Riak in the cloud.  

Customers can deploy either open source Riak or the more robustly-supported Riak Enterprise on the SoftLayer cloud.  Basho and SoftLayer have worked together to provide servers specifically tuned to run Riak and, quite interestingly, provide the servers in physical "bare metal" form, rather than as virtual cloud server instances.  While that can make the deployment more complex, the companies say the process is nonetheless automated, and that servers are provisioned in under two hours.

They grow up fast
Clearly the BI and Big Data worlds are converging now, with Hadoop and NoSQL databases acquiring the accoutrements they need to work in the Enterprise.  Before, companies in the space were, arguably, catering to what investors were impressed by. Integrating with SAS, offering bare-metal servers, cloud backup and protecting infrastructure are the things customers care about, and that's the stuff that profitability is made of.

It was only two weeks ago that Actian completed its acquisition of predictive analytics/data integration player Pervasive Software.  Add in the MPP Big Data capabilities of ParAccel, and its On Demand Integration- (ODI) based interface with Hadoop, and suddenly the sponsor of a seminal relational database product has become a hot player in Big Data and analytics.

Let's face it, the Big Data space is ripe for consolidation, so acquisitions shouldn't come as a big surprise.  But I'm not sure I would have guessed that ParAccel, whose investors include Amazon (and whose technology powers Amazon's "Redshift" data warehouse cloud service) would be acquired by the likes of Actian.

Actian, whose product portfolio also includes analytical database Vectorwise and object database Versant, now has a data-focused yet architecture-diversified array of products under one roof.

Perhaps the phenomenon of data-focused companies with multiple-decade track records acquiring newer Big Data companies will be more the rule than the exception.  A comprehensive data platform spanning relational, columnar, unstructured and predictive technologies may make the most sense, once a shakeout is said and done.

NuoDB explicitly targets cloud workloads, as the scale-out capabilities NuoDB offers align well with the elasticity characteristics of cloud computing. Under the latter, cloud infrastructure can be dynamically provisioned or de-provisioned, based on the ebb and flow of user demand, and NuoDB fits itself in just such scenarios.

Cloud via Moon(shot)

Enter project Moonshot, HP's new server initiative that focuses on high-density concentration of servers, each with very low-energy consumption. HP announced availability of its Moonshot offering, based on Intel S1260Atom processors, on April 8. With that product, a single 4.3U system contains 45 discrete physical servers.

I spoke with Seth Proctor, NuoDB's chief architect, who explained that NuoDB wanted to see how many databases it could get running on a single Moonshot box, in terms of total databases hosted on the system and the number of databases concurrently active at any given time.

Bank switching

This structure of the metrics is based on a multi-tenant hosting scenario, where multiple blogs are hosted on a single Moonshot system, and only a fraction of them encounter page view traffic in a discrete instant. The kicker: NuoDB can shut down the databases that are not in active use.

The result: NuoDB said it can serve 7,200 active databases and 72,000 total databases, on a single Moonshot system. Even under that load, the company said the server encounters only 70 percent utilization, so the numbers could theoretically go higher. NuoDB said the screenshot below shows a management dashboard readout from a Moonshot test run. The numbers are slightly lower than quoted above, but with server utilization at 67 percent, that seems OK.

Going beyond the high-density scenario is important, too. If a given blog hosted on the Moonshot system sees a significant spike in traffic, it may make sense to move that database to some bigger iron, with the option of moving it back to the Moonshot box if and when the spike subsides. NuoDB claims it can perform such a migration while the database remains live, online, and hot. The company calls this database "bursting", and if it works as advertised, it would seem a sensible companion to the high-density capability.

Patent pending

NuoDB has submitted a patent application for the bursting capability, along with the ability to "hibernate" databases when they're not in use and wake them up on demand. Proctor told me the latency involved in waking databases is only about 200 milliseconds on a Moonshot system, and significantly less on servers based on higher-powered CPUs.

Quo vadis

The NewSQL movement is intriguing, as it seeks to forge a consensus between the decades-old client-server relational database stalwarts and the bright, young NoSQL upstarts. It's not clear whether the products in this category will carve out their own long-lasting market share, or if the old-line relational vendors will adopt some NewSQL architectural ideas. Either way, it's fascinating to see innovation in the relational space, which had matured and been rather static for the first decade of this new century. Together, NuoDB and HP seem to be pushing the envelope.

During the opening keynote at the PASS Business Analytics conference today, Microsoft Technical Fellow Amir Netz and Director of Program Management for Business Intelligence, Kamal Hathi, were busy showing off the kind of data discovery work that business users can do inside Excel.  

Disclosure: I am a speaker at the PASS Business Analytics Conference and was involved in its early planning.

8-bit beginnings
Netz came to Microsoft from Israeli BI company Panorama, when the former acquired technology from the latter that would become SQL Server Analysis Services. Netz told the story, complete with photos, of growing up in Israel, being given an Apple II computer by his parents, and becoming fascinated with data and spreadsheets.  He started with VisiCalc, and eventually began running his own business doing macro programming for Lotus 1-2-3. Soon enough, Netz moved on to Excel, on the Mac.

Netz and Hathi discussed the simplicity of these tools and contrasted that simplicity with today's analytics landscape of DW, ROLAP, MOLAP, ETL, MDM, Hadoop, Hive, Pig, Sqoop, NoSQL and more. The two men explained in a tongue-and-cheek, if somewhat contrived, manner how they yearned for the data tech simplicity of earlier times.

Data idol
And so back to the spreadsheet we went, as Netz and Hathi used Excel 2013, with PowerPivot, Power View and Data Explorer, to go through reams of Billboard chart data and expose interesting factoids about popular music.  Along the way, we discovered that Rihanna has single-handedly brought Barbados into the top tier of pop music countries, and that Roxette is Sweden's best pop ensemble, as far as Billboard chart showings go. (I bet you thought it was ABBA; I know I did.) 

The last phase of the Microsoft duo's demo? An announcement that the company's project "GeoFlow," a 3D geographical data visualization add-in for Excel, is now available as public preview. GeoFlow mashes up Bing Maps and technology from Microsoft Research to render data in time-lapsed, geographic space. Netz demoed the impressive technology and, as a kicker, did so on what looked to be an 80" Perceptive Pixel touch-screen display, morphing the keynote stage into a quasi news studio.

Fun with data
Netz also talked about the importance of making data fun, and showed how visualizations built in Excel can be shared online, on Office 365. In so doing, he alluded to the same emotional, social approach to data discovery that Tableau has built its entire business on, a business it now aims to take public.

The PowerPivot, Data Explorer and now GeoFlow add-ins are available for download. Just be aware that GeoFlow requires the "ProPlus" version of Excel 2013 (via volume license, or an Office 365 subscription that provides access to that version of Excel.) Does this put a damper on Microsoft's "BI for the Masses" story? I'd say so, and I'm hoping some other folks will as well.

The title of this post is a paraphrase of a question raised, by IDC analyst Carl Olofson at an IBM Big Data analyst event earlier this week. Carl's question neatly summarized our impressions from the session, which centered around some big data announcements that IBM had made. It concerned some new performance improvements that IBM has made that might render some issues with poorly formed SQL moot. More about that in a moment.

The question was all the more fitting and ironic given the setting — the event was held at IBM's Almaden research facility, which happened to be the same place where Edgar (Ted) Codd invented SQL; IBM will video webcast excerpts on April 30.

Specifically, IBM made a series of announcements; while much of the press focused on the announcement of a preview for IBM's PureData for Hadoop appliance, to us the highlight was unveiling of a new architecture, branded BLU acceleration. Independent DB2 consultant Dave Beulke, whom we met at the launch, has published one of the best post mortems on the significance of the announcement.

BLU is supposed to be lightning fast. BNSF railroad, a BLU beta customer, reported performing a 4-billion row join in 8 milliseconds.

So what does this all mean?

Databases are assuming multiple personalities

BLU acceleration consists of a new engine that accelerates database performance. Let’s dissect that seemingly innocuous — and ambiguous — statement. Traditionally, the database and the underlying engine were considered one and the same. But increasingly, databases are evolving into broader data platforms with multiple personalities that are each designed for a specific form of processing or compute problem scenarios. Today's EMC Greenplum), run your SQL analytic queries against the relational engine, which happens to use Hadoop's HDFS as the back end file system. And for Hadoop, new frameworks are emerging alongside MapReduce that are adding interactive, graph, and stream processing faces.

This week's announcements by IBM of the BLU architecture marked yet another milestone in this trend. BLU is an engine that can exist side by side with DB2's traditional row-based data store (it will be supported inside DB2 10.5). So you can run existing apps on the row store while migrating a select few to tap BLU. BLU is also being made available for IBM's Informix TimeSeries 12.1, and in the long run, you're likely to see it going into IBM’s other data platforms (think PureData for Analytics, Operational Analytics, and Hadoop models).

From IBM: More mixing and matching

In the same spirit, we expect to see IBM (and its rivals) do more mixing and matching in the future. We’re waiting for IBM to release an appliance that combines SQL analytics side by side with an instance of Hadoop, where you could run blended analytic queries (think: analytics from your CRM system alongside social, weblog, and mobile data harvested by Hadoop).

And while we're on the topic of piling on data engines, IBM announced a preview of a MongoDB style); it will become yet another engine to sit under the DB2 umbrella. We don’t expect MongoDB users to suddenly flock to buy DB2 licenses, but it will be a way to for existing DB2 shops to add an engine for developers who would otherwise implement their own Mongo one-off projects. The carrot is that IBM JSON takes advantage of data protection and security services of the DB2 platform that is not available from Mongo.

Dissecting BLU

BLU includes a number of features that individually, are not that unique (although there may be debates regarding degree of optimization). But together, they form a well-rounded approach to not only accelerating processing inside a SQL platform, but allowing new types of analytic processing. For instance, think about applying some of the late-binding schema practices from the Hadoop world to SQL (don’t believe for a moment that analytics on Hadoop doesn’t involve structuring data, but you can do it on demand, for the specific problem).

Put another way, in the Hadoop world, the competitive spotlight currently is on convergence with SQL. And now in the SQL world, styles of analytic processing from the NoSQL side are bleeding into SQL. Consider it a case of man bites dog.

The laundry list for BLU includes:

• Columnar and in-memory processing — most Advanced SQL (or columnar engines are increasingly being incorporated alongside existing row-oriented stores inside relational warhorses. Columnar lends itself well to analytics because it reduces table scanning (you only need to look at specific columns rather than across entire rows) and focuses on aggregate data rather than individual records

• Data compression — compression and columnar tend to go together because, when you focus on representing aggregates, you can greatly reduce the number of bits for providing the data you need, such as averages, means, or outliers. Almost every column store employs some form of compression with ratios in the double-digit territory common. BLU is differentiated by a feature that IBM calls "actionable": You can read compressed data without de-compressing it first, which significantly boosts performance because you can avoid de-compress/re-compress compute cycles

• Data skipping — many analytic data stores incorporate algorithms for minimizing data scans, with BLU’s algorithms doing so by ferreting out non-relevant data.

There are more optimizations under the hood. For instance, BLU tiers active columnar data into and out of memory and/or Flash (solid state disk) drives. And while in memory, BLU optimizes processing so that several columns can be crammed into a single memory register; that may sound quite geeky, but this design pattern is a key ingredient to accelerating throughput.

IBM contends that its in-memory and Flash optimizations are "good enough" to the point that a 100% in-memory PureData appliance to counter SAP HANA is not likely. But for Flash, never say never. In our view, given rapidly declining prices, we wouldn’t be surprised to see IBM at some point come out with an all-Flash unit.

Again, what does this mean for SQL and the DBA?

Now, back to our original question: When performance is accelerated to such an extent, does it really matter whether you’ve structured your tables, tuned your database, or formed your SQL statements properly? At first blush, that sounds like a rather academic question, but consider that time spent modeling databases and optimizing queries is time diverted from taking on new problems that could cut into the development backlog. And there is historical precedent; in SQL’s early days, conventional wisdom was that it required so much processing overhead (compared to hierarchical file systems that prevailed at the time) that it would never scale for the enterprise. Well, Moore’s Law brute forced the solution; SQL processing didn't get that much more efficient, but hardware got much more powerful. Will on-demand SQL acceleration do the same for database modeling and SQL querying? Will optimization and automation make DBAs obsolete?

It seemed sacrilegious that, nearing the 40th anniversary of SQL, that such a question was posed at the very place where the technology was born.

But matters aren't quite so black and white; as one set of problems get solved, broader ones emerge. For the DBA, the multiple personalities of data platforms are changing the nature of problem-solving: instead of writing the best SQL statement, focus on defining and directing the right query, to the right data, on the right engine, at the right time.

For instance, a hot new mobile device is released to the market with huge fanfare, sales initially spike before unexpectedly dropping through the floor. Such a query might fuse SQL (from the CRM analytic system) with sentiment analysis (to see what customers and prospects were saying), graph analysis (to understand who is friends with, and influences, whom), and time series (to see how sentiment changed over time). The query may run across SQL, Hadoop, and possibly another specialized data store.

Admittedly, there will be a significant role for automation to optimize such queries, but the trend points to a bigger reality for DBAs where they don’t worry as much about SQL schema or syntax per se, but focus more on optimizing (with the system’s help) data and queries in more global terms.

Goldman, Sachs & Co and Morgan Stanley & Co LLC are acting as lead joint book-running managers for the offering. Credit Suisse Securities (USA) LLC and JP Morgan Securities LLC are acting as book-running managers. UBS Securities LLC and BMO Capital Markets Corp are acting as co-lead managers, and JMP Securities LLC is acting as co-manager.

Tableau has connectivity and partnerships with some of the biggest names in BI and Big Data. Customers love the product, and it's been on an upward trajectory for quite some time. An IPO now is certainly consistent with that track record and momentum.

It will be interesting to watch the company move beyond a "land and expand" sales approach to a perhaps more conventional Enterprise sales play. It also remains to be seen if Tableau will need to broaden its portfolio beyond its core namesake product.

Either way, this is a big deal.

MapR M3 will be bundled with Ubuntu 12.04 LTS and 12.10 via the Ubuntu Partner Archive, and MapR's M5 distribution will be certified for Ubuntu, too.

The two companies are also working together to develop a "Juju charm" to facilitate deployment of MapR into OpenStack cloud environments. The charm should be available by April 25.

As a kicker, MapR is making the source code for the component packages of its Hadoop distro publicly available on GitHub.

Pervasive Hadoop

If Hadoop is the de facto operating system for big data, then it certainly makes sense that it be part of a popular, general computing operating system, and deployable to open-source cloud computing infrastructure. And given MapR's integration in, and partnerships with, Amazon Web Services Elastic MapReduce, Google Compute Engine, and now Ubuntu Linux, the San Jose, California, company is taking a lead in embedded Hadoop trend.

When I started this blog a bit over a year ago, I felt convinced that the separation between Big Data and Business Intelligence was arbitrary and contrived. It would seem that San Mateo, CA-based Platfora agrees, as it is today releasing for general availability its namesake product that seeks to bridge the BI-Big Data divide.

To your last dying day
The Big Data world, which has orbited around Hadoop, and the BI world, which has orbited around OLAP cubes, charts and dashboards, have mutually maintained their segregated status quo.  Big Data and BI are the Jets and the Sharks of the technology industry: they live in the same neighborhood and have much in common, but thrive on obsessing over their differences.

Apache Hive has attempted to broker a peace between the warring clans, but has achieved dtente at best.  Yes, most BI tools now connect to Hadoop via Hive, but using that technology, Hadoop keeps working in its batch mode ways, to the chagrin of BI tools that want quick results, so they can fire off subsequent queries.

Special sauce
It would seem the folks at Platfora agree that a positive peace is possible, and that the mere truce brought about by Hive is at best a stopgap solution, and one that has outlived its usefulness. Platfora's namesake product is a business-user-friendly BI tool that combines an HTML 5-based user interface, a data visualization engine, and an in-memory analytical database — supporting what Platfora calls its "Fractal Cache" technology — that maintains in-memory "lenses."

Platfora is certified with the big three Hadoop distros: Cloudera's Distribution Including Apache Hadoop (CDH), the Hortonworks Data Platform (HDP), MapR, and even newcomer EMC/Greenplum Pivotal HD.  My guess is that Platfora won't take long to obtain similar certification for Intel's Distribution for Apache Hadoop as well, although no such certification has been announced.

Same thing, only different
I met with Platfora's Founder and CEO, Ben Werther, at last week's GigaOM Structure:Data event at Chelsea Piers in New York City. Given my own opinion on the futility of separating Big Data and BI, I was very interested in what Mr. Werther had to say. But I was also skeptical. After all, Cloudera's Impala, Hadapt, and numerous Massively Parallel Processing (MPP) data warehouse products, all attempt to provide interactive (rather than batch) query capabilities over Hadoop, to enable BI-style analysis of Big Data. What makes Platfora different?

I'll withhold judgement until I see the product demoed in-depth, but for now I can at least convey what Werther told me: unlike various data warehouse solutions which must be supported by careful schema design and sometimes elaborate Extract, Transform and Load (ETL) processes, Platfora supports the same sort of just-in-time schema determination as Hadoop itself. Platfora also generates its own, optimized MapReduce jobs, providing a nice middle-ground between the tedium of hand-coded MapReduce and Hive's somewhat one-size-fits-all approach of generating MapReduce code by translating SQL queries.

Hadoop for the masses
Effectively, Platfora is taking the concept of Pervasive BI and bringing it forward to the Big Data world. But the vision of Pervasive BI was never quite realized, so can Platfora make Pervasive Big Data a reality?

I think it's a steep climb, but I'm glad someone's trying. And with Werther having served as a Senior Product Manager at both Siebel and Microsoft, Director of Product Management at Greenplum and VP of Products at DataStax, Platfora's certainly coming at it from the right vantage point.

BI, SQL, and Hadoop are converging, with momentum that appears unstoppable. The only questions are how long will the convergence take, and whose solution(s) will win out.

The three products have a lot in common; they all produce excellent visualizations with a lowered authoring barrier-to-entry for business users, they also each have their own analytical engine, using some flavor of in-memory, column store technology.

The race tightens

One thing that has been a distinction for Tableau is its ability to execute queries on the data source's engine rather than its own. For queries on large data sets, this is especially important.

Tibco began to catch up with its 5.0 release, adding remote query capabilities for Microsoft's SQL Server and SQL Server Analysis Services, as well as Teradata and Oracle. Tibco 5.5 effectively closes that gap, as it now supports remote query to more than 30 data sources.

Predictive analytics, new visualizations

Spotfire 5.5 also integrates with Teradata Aster, which, through remote querying capabilities and Aster's SQL-MapReduce technology, permits in-database predictive analytics to be performed before data is visualized.

Spotfire 5.5 also supports rule-based visualizations, contextual highlighting, "visual joins" for heterogeneous data mashups, and map visualizations for North America, South America, Europe, the Middle East, and Asia. Tibco told me that map support for Africa and Australia will be added in the future.

It's the enterprise, stupid

It's important to keep in mind that, unlike BI pure-plays Tableau and QlikTech, Spotfire is on offer from Tibco, a major force in enterprise software. As such, Spotfire connects with Tibco's DataSynapse GridServer and BusinessEvents products.

This buttresses the predictive analytics capabilities to cover operational, business-to-business, and enterprise events and transactions. And to optimize such scenarios, Tibco now offers a free developer edition of the Tibco Enterprise Runtime for R (TERR), which provides for massively parallel statistics processing on Tibco GridServer.

As Big Data sees more enterprise adoption, the importance of enterprise data sources gains significantly more importance. And as Tableau and QlikView gain in popularity, the continued viability of a third horse gains in importance as well.

Related stories

• Tableau 8 unveiled: Can it keep the good times rolling?

• Gartner, IBM, Teradata make Big Data announcements

Analytics is fairly new to sports but is nothing new to business. With today's technology, vast amounts of data is analyzed by increasingly more powerful computers to predict success rates for game strategies, a player's potential for success, betting, or marketing a team. It's all there right in front of you. The field, made popular in sports by statistician Bill James and Oakland A's general manager Billy Bean, the focus of the book and movie Moneyball, is based on crunching numbers and data over watching an athlete's or team's actual performance. Both men have been featured guests at this conference.

The goal is to make a team better while using fewer resources. It helps a team pick important role players at a lower cost while avoiding the ones who demand higher salaries but may provide a low return on a team's investment. Even small market teams can be competitive — case in point the Oakland A's.

We'll check out strategies put forward by sports management and researchers that are based on cold numbers, not hunches or out-of-date game plans.

A paper presented at this conference tackles when and where and when an NFL coach should send out his field goal unit. Here's the analytical analysis presented by three students from MIT's Aueronautics and Astonics Department — Torin Clark, a PhD candidate, and two graduate students, Aaron Johnson and Alexander J. Stimpson. Their study is one of one of eight finalists in the research-paper competition at this year’s Sloan conference.

Based on their examination of 11,896 NFL field goal attempts, they've determined that environmental factors are much more important than psychological factors in the success of a field goal attempt. Calling a timeout to ice a kicker has little value while factoring weather conditions such as wind velocity or temperature are much more critical.


Photo: ZDNet.com

Brian Burke who authors a blog, Advanced NFL Stats, produced detailed analysis, "Fourth Downs in the New Overtime: First Possession." In an NFL overtime game, the team that possesses the ball first can only win the game with a touchdown or safety. After that possession, any score will win the game.

The numbers show the team which has the ball first in overtime has a better chance of winning if, on fourth down deep in its own territory, it tries for a first down instead of punting. The deeper the team is in its own territory, the chances are better that that the team will make a first down rather than give the ball to the opposing team which should have great field position and is likely to score. Over the long run, this strategy may win more games on average but an NFL coach is looking at losing his job if it doesn't work just once.

Another finding is that long field goals should not be attempted on a team's first possession in overtime. Their projected success rate is overshadowed by the loss of field position if it is missed and even if it is good, the opposition still has a chance to win.

Photo: CBSNews.com

Why do the best football (soccer) players appear to have better field vision than others? To answer that question, Geir Jorde of the Norweigan Sports Institute, Jonathan Bloomfield of Hull University, and Johan Heijmerikx of the University of Groningen, Netherlands examined 1,279 close-up videos of more than 118 midfielders and forwards in the Barclay's Premier League.

The study focused on how the players use head and body movements to help them see the field better and make better split-second decisions. The most important finding was that players, especially midfielders, who better explore the field will make more accurate passes — something that managers, scouts, and fans usually overlook. Here's the full study.

Photo: Wikipedia

In his new book, Sports Analytics: A Guide for Coaches, Managers, and Other Decision Makers, Benjamin Alamar says that sports analytics is in its infancy and teams can gain a significant advantage by using it. He cites a recent survey which showed that while 37 percent of teams have easy access to one another's data, 37 percent of them do not employ a database programmer.

Alamar says that sports and business need analytic tools and computer power to sift through massive amounts of data to produce reports that can be used to develop a competitive edge. He refers to how small market teams such as the Oakland A's have used sports analytics to successfully compete with the larger, better financed organizations.

Photo: Wikipedia

Chad Millman, Editor in Chief of ESPN The Magazine, tells how sports analyst Mike Wahl has examined sports betting from many angles and found an almost sure winner in college football. Typical of many current sports gurus, Wahl earned an MBA and then worked in business as a financial analyst before switching to sports.

To find the right winning bet, Wahl searched through six years of college football games (376) when a team was favored by 20-25 points. To bet on an outright game winner is supposed to be evened out by having the wager on the favorite cost more than the one on the underdog. He found that if you bet the same amount on the favorite as an outright winner, you'd have won overall for six years in a row and your return would have been 12.24 percent. But I'm sure this loophole will be closed very soon.

Billy Beane changed the face of baseball by using new statistics, such as on-base percentage and slugging average, as a better indication of a player's vaue than traditional baseball measurements — batting average, stolen bases, and RBI. An even newer analysis of a baseball player's value is called WAR (wins above replacement) which attempts to indicate how much a player contributes to his team. One major leaguer who stands out is Atlanta Braves outfielder Jason Heyward. His three-year WAR rating is the sixth highest for outfielders under 22 since 1961.

For more read:  ESPN Stats and Info. There are a lot of interesting blogs on the site. The Atlanta Braves analysis is third on the list.

Photo: Wikipedia

The English Rugby Union's frequent champion Leicester Tigers are using IBM's predictive analytics software to assess injury risks and then deliver training programs for players at risk. The Tigers are hoping analytics can keep players on the field longer.

IBM has developed software which is designed to measure fatigue levels and game intensity. The Tigers will also crunch physical and biological data from its 45 players. In addition, the Tigers plan to use big data to measure psychological factors such as stress levels, social issues and environmental stress.

IBM's software will also be used to gauge the performance for its under-19 academy feeder teams and choose players accordingly.

Caption: Larry Dignan

Photo: Wikipedia

Kevin Mongeon is the principal owner at The Sports Analytics Institute and shows how sports analytics can impact on winning and losing in his blog, "More Hockey Data"? Unlike baseball where specific actions show measureable results, hockey is played in a continous flow making the game more difficult to put into an analysis on paper.

Mongeon says additional data is needed to discover a statistical path to a winning season. He needs statistical models that can examine a player's abilities even under different scenerios.

Photo: Wikipedia

An NBA study by Jenna Wiens, Guha Balakrishnan, Joel Brooks, and John Guttag from MIT examines the offensive/defensive strategy about whether it's better to crash the boards for an offensive rebound or lay back and play defense.

This detailed analysis developed the Crash Index and Retreat Index to determine which philosophy gives a team the opportunity to score more points. The study found that when a team made a big effort for the offensive rebound, it gained more than passivly staying back on defense. The study does note that it does not take player personel into account.

Credit: Wikipedia

John Parolin, Statistics Analyst, ESPN Stats and Analysis is part of a team that recorded every single play in the NFL 2012 regular season and playoffs. For this year's Super Bowl, the easy finding was that major mid-season changes, offensive coordinator for the Ravens and quarterback for the 49ers, led both teams through the playoffs. The Ravens rush/pass ratio turned from 40 percent to 49 percent after the change, while the 49ers new quarterback, Colin Kaepernick, experienced great success with the zone-read option where he determined the play based on the actions of an unblocked linebacker.

ESPN found that the one team, Atlanta Falcons, had overplayed Kapernick's running ability in the zone-read option, and held him to just 21-yards rushing — and almost led them to an upset of the 49ers. The Ravens successful defense of the zone-read option, in the first half anyway, was one of the keys to their victory.

Photo: Wikipedia

Damien Demaj, Geospatial Product Engineer at ESRI analyzed the Olympic Gold Medal tennis match between Roger Federer and Andy Murray. He studied "the spatial variation of serve patterns" in his project, "Using Spatial Analytics to Study Spatio-temporal Patterns in Sport."

Demaj's analysis focused on the placement and bounces of each serve in the match. He found that the location where the server was standing, the service patterns, and the importance of that particular point in the match were keys to understanding the game. For example, in the ad court Federer's spacial service cluster went left most of the time with a wide spread while in the deuce court he was more accurate. Murray's clusters were more focused and favored the right side of the court. Murray won the match: 6-2, 6-1, 6-4.

Talk about detailed analysis, here's one of the tools he designed, "The sequence of bounces then allowed us to create Euclidean lines between p1 (x1,y1) and p2 (x2,y2), p2 (x2,y2) and p3 (x3,y3), p3 (x3,y3) and p4 (x4,y4) etc in each court location."  

Credit: Wikipedia

Ed Feng, founder of ThePowerRank.com, tells how to make it in the sports analytics world. Any number of people can sift through data on their own computers but few are able to go to the next step and find a full-time job in sports analytics.

While expertise in numbers crunching and being adept with the use of social media will help, Feng says that there's no substitute for "real human interaction." That means being able to shake someone's hand or looking at them straight in the eye, according to Feng. He suggests attending conferences such as the Sloan Sports Analytics Conference where you can show off your stuff.

Credit: iStockphoto.com

Background considerations
Analytics functionality is most useful when it's integrated into vertical market and line-of-business applications.  While technologies like Hadoop and R are terrific on their own, and while their integration with analysis and data discovery clients, from Excel to Tableau, is even better, embedding analytics functionality in operational applications is what makes it most effective.

In certain industries where use of data is, or is becoming, culturally ingrained, like health sciences, the integration and embedding of analytics technology into operational applications is gaining traction.  And the resulting gravitational pull is what's behind the merger of IMS and Appature.

Separate corners
IMS has long been in the business of procuring and aggregating depersonalized healthcare data, including prescription transactions, patient profiles, provider profiles, and insurance claims, and making that data available for analysis.

 management for pharmaceutical companies, along with the data acquisition, cleansing and analytics necessary to monitor those campaigns in real-time and optimize the design of subsequent campaigns.

Pharma marketing's active ingredient: data
The folks I spoke with at Appature tell me that IMS manages over 17 petabytes of data which, they say, is more data than the IRS has.  Whether or not the last bit's true, 17PB is certainly a lot of data, and the company has been tracking this data for a long time.  

That data is more important now though.  Pharmaceuticals marketing traditionally has been carried out by individual reps in the field, working one-on-one with doctors, in their offices.  But managed care has made doctors busier, leaving less time for such meetings, and new regulations limit reps' interactions with doctors in any case.  So direct marketing, optimized by analytics, has become indispensable and Pharma tech marketing dollars are shifting from CRM and ERP to analytics.

2+2
With all that in mind, why not take Big Data that's been tracked for quite some time and leverage it from Appature's more recent-vintage SaaS Big Data analytics platform?  That makes a lot of sense and underscores why Big Data is more than fad; in reality, it's a new brand for something that's not at all brand new.  This merger seems based on that very fact.