Business Differentiation through Machine Learning.

I look at my child and marvel as he embraces the ever fast moving world around him, adapting to new experiences, grasping technology, absorbing a bombardment of information from so many sources.  It’s staggering to watch their progress from basic learning of just accepting facts they are taught, to augmenting those facts with their own knowledge, to asking questions, using their knowledge to express their opinions and values to others, then challenging facts and hypotheses that they once accepted to adapting their knowledge, understanding, decisions and value systems. And then they start reasoning with their parents! Their brains are like sponges using their senses of seeing, hearing, touch, smell and taste, absorbing information, experiences, emotions, constantly connecting new events and situations with those from the past – pushing boundaries and testing us.

Watching my son learn was fascinating but quite frankly businesses can’t wait years to learn about their data and how best to act on it.


From Business Intelligence to the Intelligent Business.

The business world is complex with many fast moving dynamics.  We only see the tip of the iceberg when it comes to the total information that exists at any point in time as it grows exponentially. Nor are we able to fully harness the collective thoughts or full knowledge of the people around us.  This is where cognitive technologies such as the many forms of machine learning (ML) can really be a huge advantage to business people.

Simply put machine learning is the capability of computers to learn without being explicitly programmed.

Imagine systems that have total information awareness. “Nodes” that connect with each other on premise and in the cloud that learn about each other like synapses in the brain.  Harnessing the collective consciousness of these “machines” (don’t restrict your thoughts to hardware here) and applying intelligence to advise on the “optimal” decisions resulting in the “optimal” outcome is what every organization would want. I think I just defined the killer-app for business.

What does it look like?  Well I envisage it as the graphic below – a cycle or workflow of constant learning.

Screen Shot 2016-06-29 at 8.34.45 AM

Figure 1 : Machine Learning  – a cycle of constant learning


What’s out there today?

The good news is that much of these capabilities exist in IBM solutions today and much more in our research labs yet to emerge and potentially change the industry.

So how do you get started?  There are four main states when considering knowledge: Know what you know, Know what you don’t know, Don’t know what you know, Don’t know what you don’t know.  And all this starts with your data. We have solutions that can catalogue what information you have, discover information that you didn’t know you had and help identify information that is missing or untrustworthy.  These can help reduce not knowing what you don’t have.

Next you need capabilities that can ingest information where ever it is without having to move it.  From this data we can apply machine learning (ML) that can discover relationships, rationalize, predict what might happen next and actually become intelligent (all knowing) about the data and all previous outcomes.  This can only be achieved through continuous learning and adapting. There are also other steps around optimization of that learning that can be read here. There are many vendors that claim to provide one form of ML capability or another and while that may be interesting in its own right, on its own it has minimal value.  Combining all forms or learning and analytics can take you closer to the bigger picture of “cognitive” in which IBM strives to be a leader. I’ll expand a little more on cognitive later in this blog.


From Tic-Tac-Toe , Jeopardy, Crime Prevention, Cancer Treatments and more.

In my early student days I learned how to code simple recursive algorithms using rote learning to produce a game of Tic-Tac-Toe that could not be beaten after five or six games. It did this by minimizing its losses by avoiding outcomes that would lead to failure.  In 2011 “Watson” competed on Jeopardy and beat the top contestants.  It had access to 200 million pages of structured and unstructured content consuming terabytes of disk storage including the full text of Wikipedia but was not connected to the Internet during the game. For each clue, Watson’s three most probable responses were displayed on the television screen. Watson consistently outperformed its human opponents on the game’s signaling device, but had trouble in a few categories, notably those having short clues containing only a few words – kind of similar to human language ambiguity.  Its natural language processing, predictive scoring and models were key to its success.

In February 2013, IBM, Wellpoint and Memorial Sloan-Kettering Cancer Center announced the first commercially developed Watson based cognitive computing technology to be implemented for utilization management recommendations to physicians in lung cancer treatment at Memorial Sloan Kettering Cancer Center in conjunction with health insurance company WellPoint Inc. (now Anthem Inc). At the time of writing this blog, nearly 43,000 organizations have registered to use IBM Watson Healthcare Analytics Platform .(1)


Machine Learning 101

I am often asked “How confident are you in that decision?” I used to base my answer on the strength of the information I had – and some of it was intuition, gut feel, life experiences.  But now I can scientifically put a figure on it – as precise as the underlying information allows of course.  In fact, well established IBM analytics products have been using these predictive models and scoring algorithms (included in Watson Analytics) across many industries for years to better manage risk, and identify potential fraud (I’ll tell you about my credit card experience in a later blog while attempting to rent a vehicle).  In 2015 IBM donated its SystemML to the Apache™ Software Foundation. SystemML is a flexible machine learning system designed to auto-scale to Spark and Hadoop® clusters and extends the core machine learning in the Apache Spark™MLlib libraries. We also have machine learning in many other forms available as-a-service including but not limited to Natural Language Classifier, Retrieve and Rank, AlchemyVision –and many others that we will describe in more detail later.   Below is a diagram of how machine learning is implemented as a generic model.

Screen Shot 2016-06-29 at 8.34.04 AM

Figure 2 – A generic machine learning model


Beyond Traditional ML – Learning through Senses.

As mentioned earlier IBM has many other forms of machine learning technologies that help  differentiate our cognitive capabilities from other vendors. Using vision, language and other ML technologies begins to more closely simulate human behavior of understanding, reasoning and learning.  Below are some of the key ML technologies that are being used by many organizations around the world.

AlchemyVision is an API that can analyze an image and return the objects, people, and text found within the image. AlchemyVision can enhance the way businesses make decisions by integrating image cognition. Organizations across a variety of industries ranging from publishing and advertising to eCommerce and enterprise search can effectively integrate images as part of big data analytics being used to make critical business decisions by better targeting ads, organizing image libraries, improve consumer experience, monitor your brand, profile target markets, improve researching.  The Tabelog case study is particularly interesting. Over 40,000 foodies visit the Tabelog site, confident that Tabelog will provide accurate and reliable recommendations. Additionally, more than 200,000 registered restaurants use the site to help brand and promote their establishment. Try it now

Natural Language Classifier is a service that enables developers without a background in machine learning or statistical algorithms to create natural language interfaces for their applications. The service interprets the intent behind text and returns a corresponding classification with associated confidence levels. The return value can then be used to trigger a corresponding action, such as redirecting the request or answering a question. The Natural Language Classifier is tuned and tailored to short text (1000 characters or less) and can be trained to function in any domain or application. Typical usage scenarios are:

  • Tackle common questions from your users that are typically handled by a live agent.
  • Classify SMS texts as personal, work, or promotional
  • Classify tweets into a set of classes, such as events, news, or opinions.
  • Based on the response from the service, an application can control the outcome to the user. For example, you can start another application, respond with an answer, begin a dialog, or any number of other possible outcomes.

You can try it out by clicking here.

Retrieve and Rank is a service helping users find the most relevant information for their query by using a combination of search and machine learning algorithms to detect “signals” in the data. Built on top of Apache Solr, developers load their data into the service, train a machine learning model based on known relevant results, then leverage this model to provide improved results to their end users based on their question or query. The Retrieve and Rank Service can be applied to a number of information retrieval scenarios. For example, an experienced technician who is going onsite and requires help troubleshooting a problem, or a contact center agent who needs assistance in dealing with an incoming customer issue, or a project manager finding domain experts from a professional services organization to build out a project team.  You can try it out here.


From Crystal Ball Predictions to Prescriptive Actions.

So predicting a hurricane or an outcome or recognizing images and video or understanding the importance of a particular piece of information is just a first step. Now you need to take prescriptive actions on that understanding to, for example, survive the hurricane or achieve business advantage from a business opportunity that may only last a short time.

IBM has the capabilities not only to help provide an advanced and wide range of machine learning capabilities and to act on them – but to do it in the “right-time” – for example being able to predict whether a trade or bank transaction is legitimate or fraudulent 15,000 times a second.  In business, speed is of the essence.


Cognitive + Cloud = Optimal Business Outcomes

So it seems that machine learning can be advantageous (even smarter) and faster than humans in certain situations – given complete and accurate data. Combining the wide range of advanced machine learning capabilities described above with the ability to act prescriptively on what has been learned with IBM’s full cognitive analytics capabilities can yield many opportunities to potentially out-maneuver your competition, act with confidence and help your organization become the optimal Intelligent Business. Cloud is key here because it has the potential for everyone and everything  (including data) to be interconnected.

Machine learning can help enable cognitive systems to learn, reason and engage with us in a more natural and personalized way. These systems will get smarter and more customized through interactions with data, devices and people. They will help us take on what may have been seen as unsolvable problems by using all the information that surrounds us and bringing the right insight or suggestion to our fingertips right when it’s most needed. Over the next five years, machine learning applications could lead to new breakthroughs to help amplify human abilities, assist us in making good choices, look out for us and help us navigate our world in powerful new ways.

In summary machine learning in all its forms has the potential to bring the collective knowledge and consciousness of humans and machines together to help make the world a better, safe place.

In the following months the team and I will be taking you on a journey exploring many more aspects of IBM’s cognitive and learning capabilities.

Of course you won’t be able to predict where I’m taking you next.   🙂

For more information on IBM’s cognitive and machine learning capabilities click this link


Dinesh Nirmal,

Vice President, Development, Next Generation Platforms, Big Data & Analytics

Follow me on Twitter @DineshNirmalIBM


Jean Francois Puget (PhD),

Distinguished Engineer, Chief Architect IBM Analytics Solutions

Follow me on Twitter @JFPuget


Foot notes

(1) IBM Watson Analytics Team based on number of registrations as at June 27 2016

TRADEMARK DISCLAIMER: Apache, Apache Hadoop, Hadoop, Apache Spark, Spark are trademarks of The Apache Software Foundation.



Why a Cloud First Strategy Can Benefit Customers with IBM BigInsights on Cloud.

Hadoop – the early years

The origins of Apache™Hadoop® go as far back as 2003 in reference to the emergence of a new file system, followed by the introduction of MapReduce and the birth of Hadoop in 2006. It achieved notoriety and fame as the fastest system to sort a terabyte of data and when it became an Apache open source project (Apache Hadoop) it sent a signal that it was ready for prime time.  The world never looked back.  Within IT shops and even board rooms there was huge interest, excitement – even hype with suggestions that it might replace the enterprise warehouse.

A quick Hadoop refresh

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Maturity, BigInsights platforms and the move to cloud

All technologies move through a maturity cycle. Hadoop is no exception and is maturing fast. IBM® saw the opportunity to build enterprise ready mission critical Hadoop based solutions delivering its BigInsights™ portfolio which adds significant value to the core Apache Hadoop open source software. IBM helped lead and drive the Open Data Platform initiative (ODPi) [see] to encourage interoperability across different Hadoop vendors.

Built around a no-charge open source based core which includes Apache Spark™ called the IBM Open Platform with Apache Hadoop (IOP), IBM BigInsights brings a rich set of capabilities from advanced and high performance analytics such as BigSQL, to visualization through BigSheets all neatly brought together to meet the needs of different personas.  IBM BigInsights is cited as being a leader in The Forrester Wave™: Big Data Hadoop Distributions, Q1 2016 available from Forrester.

Usage of Hadoop in general varied widely across customers – some having multiple thousand node clusters and others just 5 or 10. Having 100s or thousands of nodes might be off-putting for some customers in terms of capital and management costs.  With BigInisghts having established itself as a leader and with IBM focused on a Cloud First Strategy, we saw the opportunity to help customers reduce these capital and management costs, to enable them to focus on running the analytics for business advantage while providing BigInsights on a dynamic elastic and scale out infrastructure in the cloud through IBM SoftLayer and Bluemix technologies from any of our many data centers around the world.

Screen Shot 2016-05-16 at 2.10.37 PM

Figure 1: IBM Open Platform and BigInsights – cloud services.


The following report cites IBM as a leader : “The Forrester Wave™ : Big Data Hadoop Cloud Solutions, Q2 2016” which states :

“IBM differentiates BigInsights with end-to-end advanced analytics. IBM BigInsights runs atop IBM’s SoftLayer cloud infrastructure and can be deployed on any of 17 global data centers. IBM’s client relationships require it to be flexible in how it offers Hadoop in the cloud and offer highly customized configurations. IBM is making significant investments in Spark, offering data science notebooks that run with the platform. Enterprises using IBM’s data management stack will find BigInsights a natural extension to their existing data platform. The company has also launched an ambitious open source project, Apache SystemML, from its newly minted Spark Technology Center. IBM’s customers value the maturity and depth of its Hadoop extensions, such as BigSQL, which is one of the fastest and most SQL-compliant of all the SQL-for-Hadoop engines. In addition, BigQuality, BigIntegrate, and IBM InfoSphere Big Match provide a mature and feature-rich set of tools that run natively with YARN to handle the toughest Hadoop use cases.”

The report shows IBM scored among the highest in the solution configuration, data security, data management, development, cloud platform integration, ability to execute, road map, professional services, fixes and partnerships criteria.


In closing…

To conclude, there has never been a better time to invest in your BigInsight projects whether on-prem or in the cloud. The IBM Cloud First strategy is helping customers better manage their costs and focus on delivering business value and insight.  IBM can help abstract the complexities of managing infrastructures in a highly performing, highly available, security-rich and elastic scale-out environment across 17 worldwide multi-tenant data centers.  IBM BigInsights, combined with making data easy and our leadership and investment in Apache Spark, is helping deliver a next generation analytics platform capable of advanced analytics, machine learning, streaming, powerful SQL, graph analytics and more.

For more information on IBM BigInsights or to get started on BigInsights on Cloud click here.

Dinesh Nirmal – Vice President, Next Generation Platform, Big Data & Analytics on z

Follow me on Twitter:  @IBM_Innovation



TRADEMARK DISCLAIMER: Apache, Apache Hadoop, Hadoop, Apache Spark, Spark and the Spark logo are trademarks of The Apache Software Foundation.

IBM, IBM BigInsights, BigInsights are trademarks of the IBM Corporation.