Big Data is the Answer – What was the Question?

The Big Data Analytics promise: enable “data monetization” through more timely, more accurate, more complete, more granular, more frequent decisions. So, what exactly are the types of business problems big data analytics likely to solve?  For this you need a mini-MBA in Big Data Use Cases.

Use cases described in this posting are meant to stimulate ideas of how to apply iterative big data analytics in your own organization and enable your own analytics center of innovation.

First let’s define what makes data Big.

Big Data, Little Data

Unless you have been living in a cave you probably have heard about Big Data. We live in a world of data: transactions, feedback, and realtime interaction with customers, partners, suppliers, and employees.  

In addition to brick, click and mobile transactions, the new variable in the mix is Human generated data – explosive growth of blogs/reviews/messages/emails/pictures.  The Twitter firehose alone generates 10s of millions of tweets per day and is growing rapidly. Social graphs such as product recommendations based on circle of friends, jobs you may like (linked in), products you have looked at, people who are your contacts etc also create “second order” data  that can be mined for sentiment analytics on products or companies or fact discovery.

Another new variable is computer generated data. Computers generate data as byproduct of interacting with people or other with other device.  More interactions,  more data. This data comes in a variety of formats from semi-structured log files to unstructured binaries. This “exhaust fumes” of data can be extremely valuable.  It can be used to understand and track application or service behavior so that we can find patterns, errors or suboptimal user experience.  We can mine it for statistical patterns and correlations to generate insights.

However, if you listen to the hype, companies can harness this information learn faster, make better decisions, and stay one step ahead of their competitors.  Unfortunately, harnessing big data (and separating the signal-from-noise) is trickier than it looks.  It takes a lot of skill and superb understanding of use cases.

Big Data Use Cases

The key to exploiting Big Data Analytics is focusing on a compelling business opportunity as defined by a use case — WHAT (What exactly are we trying to do?)

Use cases are emerging in a variety of industries that illustrate different core competencies around analytics.

bigdata-use-case-1

 

Figure below illustrates some Use Cases along two dimensions:  data velocity and variety. A Use Case provides a context for a value chain:  how to move from  “Raw Data -> Aggregated Data -> Intelligence -> Insights -> Decisions -> Operational Impact -> Financial Outcomes -> Value creation.”

big_data_analytics_use-cases

Source:  SAS and IDC

E-tailing – E-Commerce – Online Retailing Use Cases

  • Recommendation engines — increase average order size by recommending complementary products based on predictive analysis for cross-selling.
  • Cross-channel analytics — sales attribution, average order value, lifetime value (e.g., how many in-store purchases resulted from a particular recommendation, advertisement or promotion).
  • Event analytics — what series of steps (golden path) led to a desired outcome (e.g., purchase, registration).
  • Right offer at the right time

Interesting Use Case – Amazon Will Pay Shoppers $5 to Walk Out of Stores Empty-Handed

Interesting use of consumer data entry to power next generation retail price competition…. Amazon is offering consumers up to $5 off on purchases if they compare prices using their mobile phone application in a store.  The promotion will serve as a way for Amazon to increase usage of its bar-code-scanning application, while also collecting intelligence on prices in the stores.

Amazon’s Price Check app, which is available for iPhone and Android, allows shoppers to scan a bar code, take a picture of an item or conduct a text search to find the lowest prices. Amazon is also asking consumers to submit the prices of items with the app, so Amazon knows if it is still offering the best prices.  A great way to feed data into its learning engine from brick-and-mortar retailers.

This is an interesting trend that should terrify brick-and-mortar retailers. While the real-time “Everyday Low Price” information empowers consumers, it terrifies retailers, who increasingly are feeling like showrooms — shoppers come to to check out the merchandise but ultimately decide to walk out and buy online instead.

Retail/Consumer Use Cases

  • Merchandizing and market basket analysis.
  • Campaign management and customer loyalty programs.
  • Supply-chain management and analytics.
  • Event- and behavior-based targeting.
  • Market and consumer segmentations.

Big Data Analytics History: In the 1980’s, the CPG / Retail industry transition from bi-monthly audit data to scanner data changed the dynamics of the industry.  RFID is transforming the supply chain.  Combining data from a diverse set of data resources—  adding customer sentiment data, geolocation data, customer preference analysis, and market trending information based on contextual text analytics, all add up to provide a level of sophistication previously unavailable to the industry.

Predictive analytics is well understood by the retail industry.  Retailers want to predict factors that might be important for a buyer to make a purchasing decision before that product ever was stocked on shelves.  What if retailers could know exactly what market dynamics were modifying demand curves before they occurred? Imagine the impact on operational efficiencies in terms of inventory cost control, intelligent distribution and routing, and demand projection.  The retail Use Cases are quite varied.

Financial Services Use Cases

  • Compliance and regulatory reporting
  • Risk analysis and management
  • Fraud detection and security analytics
  • CRM and customer loyalty programs
  • Credit risk, scoring and analysis
  • High speed Arbitrage trading
  • Trade surveillance
  • Abnormal trading pattern analysis

Risk Modeling Use Case — A large financial institution took separate data warehouses from multiple departments and combined them into a single global repository in Hadoop for analysis. The bank used the Hadoop cluster to construct a new and more accurate score of the risk in its customer portfolios. The more accurate score allowed the bank to manage its exposure better and to offer each customer better products and advice.

Trade Surveillance Use Case — A large investment bank combines data about the parties that participate in a trade with the complex data that describes relationships among those parties and how they interact with one another. The combination allows the bank to recognize unusual trading activity and to flag it for human review.

Web & Digital Media Services Use Cases

  • Large-scale clickstream analytics
  • Ad targeting, analysis, forecasting and optimization
  • Abuse and click-fraud prevention
  • Social graph analysis and profile segmentation
  • Campaign management and loyalty programs

Clickstream Use Case –  Big Box Retailer need to analyze their clickstream – §3.5 billion records; 71 million unique cookies; 1.7 million targeted ads required per day.   The problem: how to improve Return on Ad Spend (ROAS).  Also how to speed up the analytics so consumers get more relevant ads quicker, which is especially important during holiday seasons!

Suggestion Use Case.  Yelp is growing rapidly and with more than 50 million of monthly visitors and 18 million or reviews the company generates about 400GB of data a day.  That data needs to be processed and analyzed.  A simple use case is Spelling Suggestions.  By looking at millions of misspelled words Yelp uses an algorithm to create suggestions for common misspellings.  By looking at typical queries, yelp can list common suggestions for a query even before you finish typing. This is possible because Yelp analyses all the web logs from their websites

Government Use Cases

  • Fraud detection
  • Threat detection
  • Cybersecurity
  • Compliance and regulatory analysis
  • Energy consumption and carbon footprint management

Threat detection Use Case : A federal law enforcement agency has a challenge: how does it monitor threat (or criminal) behaviors and communications in order to raise awareness of interdiction opportunities while also exposing non-obvious relationships between  terrorist actors/agents?  Using massive amounts of data from numerous data sources, how could the agency proactively engage and disrupt planned criminal activities?  And how can the agency execute on these needs in a cost effective and efficient manner—exposing information to partner organizations as appropriate.

New Applications

  • Sentiment Analytics
  • Mashups – Mobile User Location + Precision Targeting
  • Machine-generated data, the exhaust fumes of the Web

socialgraph

Online Dating Use Case:  A leading online dating service uses sophisticated analyses to measure the compatibility between individual members, so that it can suggest good matches for a potential relationship.

Big Data Analytics helped customers find romance.  The algorithms that power Match.com are not very different than those that are behind LinkedIn.

Social Gaming:  Zynga in their S-1 filing claimed that “they  process and serve more than a petabyte of content for  players every day, a volume of data that is unmatched in the social game industry. We continually analyze game data to optimize our games. We believe that combining data analytics with creative game design enables us to create a superior player experience.”

Health & Life Sciences Use Cases

  • Health Insurance fraud detection
  • Campaign and sales program optimization
  • Brand management
  • Patient care quality and program analysis
  • Supply-chain management
  • Drug discovery and development analysis

Telecommunications Use Cases

  • Revenue assurance and price optimization
  • Customer churn prevention
  • Campaign management and customer loyalty
  • Call Detail Record (CDR) analysis
  • Network performance and optimization
  • Mobile User Location analysis

Telecom Use Case:  A large telecommunications provider analyzed call logs and complex data from multiple sources. It used sophisticated predictive models across that data to predict the likelihood that any particular customer would leave. The result:  Analytics helped the telecommunications company build more insightful customer relationships and reduce churn.

Utilities Industry  Use Cases

Utilities run big, expensive and complicated systems to generate power.  Each grid now includes sophisticated sensors that monitor voltage, current, frequency and other important operating characteristics.  Efficiency now means paying careful attention to all of the data streaming off of the sensors.

Utilities are now leveraging Hadoop clusters to analyze generation and consumption data via smart meters.

Smart meters  –  The rollout of smart meters as part of the Smart Grid adoption by utilities everywhere has resulted in a deluge of data flowing at unprecedented levels.   Most utilities are ill-prepared to analyze the data once the meters are turned on.

So, What’s the Big Deal?

The big deal is that if analytics is done well as shown in the figure there is room for margin expansion and additional profit.

bigdata-use-case-2

Summary

Big Data is full of valuable, unanswered questions!  The challenge is separating the actual predicitive indicators –  signal from the noise — in the data.

Companies who compete on analytics and delivering data-driven services tend to iterate quickly on big data.  This enables rapid data exploration to identify unknown relationships and trends to create new products and services.

Data overload is going to be a huge challenge for businesses and a headache for decision makers.  Public and private sector corporations are going to drown in data — from sales, transactions, pricing, supply chains, discounts, product, customer process, projects, RFID smart tags, tracking of shipments, as well as e-mail, Web traffic and social media. Without a smart use case strategy, a lot of this data will be wasted.

2. A business use case describes what the process does. It is meant to describe in technology-free jargon the business process that is used by its business actors (people or systems external to the process) to achieve their goals . The business use case will describe a process that provides value to the business actor.

Further Reading

1) Saama vividSENSE for a Saama’s point of view presentation on Big Data and why you need to listen to your Big Data.

2) CIO Analytics: Managing Business Value of IT for a whitepaper on how to measure, Plan and Cost Business Value of IT using a Service Oriented Delivery Model.

3) Big Data is the Answer – What was the Question? for a recorded version of the webinar hosted on February 17, 2012

11 replies
  1. michael.hummel@parstream.com'
    Michael Hummel says:

    Thanks to Shirish for that great article about use cases of BigData!

    I would love to add some thoughts that support his claim of BigData uses cases being real and only to takle with BigData soltuions.

    This is because not only the data changed over the past years, but also the expectations of the users and the infrastructure.

    Let’s just pick one use case… Web Analytics or Click-Stream analytics.

    In the past most companies looked at their traffic figures once a week, mostly Monday morning for the past week.
    -> infrequent, large queries on historic data, reporting-style queries, response time below 10mins acceptable

    Now-a-days shop owners want to know what happend in the past minute in comparison to a normal day at this point in time. They segment not only be time but by region, product, user-profile etc.
    They run ad or email-campaings and want to monitor the effect whilst the campagin is still running! They want to change their website 100 times a day to try out diffent layouts and their effect on conversion…
    -> not reporting, its about gaining insights and thats an iterative process
    -> many queries, expectation of response time to be in the one digit seconds or below
    -> real-time segmentation of clicks in n-dimensions down to a single user / user group
    -> up-to-date data, every clicks needs to find its way into the analytical results within a few seconds, not next day!
    -> and you will have many users working interactively with the web analytics system, i.e. high throughput capability is essential.

    Fact: the world is not only downing in data volume. Through more competition and maturity of markets the business requirements and expectations grow rappidly.
    Being able to Analyse BigData in real-time, with low-latency with high-throughput enables sustainable business differentiation.

    Be part of the change. Or the change will pull you apart.
    Best Mike

    Reply
  2. mohan@TrendwiseAnalytics.com'
    Mohan Kumar says:

    Great article Ravi and Shirish! Excellent use cases. Of course most of these are focussed on US. Would be interesting to know how Big Data is being adopted in emerging markets specially India and China.
    Thanks,
    Mohan

    Reply
  3. guido.pezzin@mail.com'
    Guido says:

    Thanks, very interesting, including self-explaining pictures. Still missing a little bit the business cases in financial services and banking. Thanks Guido

    Reply
  4. mark@chmarny.com'
    Mark Chmarny says:

    Really good read, thank you. The pragmatism of Big Data is something I have been after for a while now. While we certainly need more innovation in basic tooling, the bigger need is to demonstrate a very practical and real-life based application of big data analytics. I even went as far as building 3 high level use-cases for such applications… my 3 big data killer apps if you wish ;)

    http://mark.chmarny.com/2012/02/three-killer-big-data-app-opportunities.html

    Love to hear you thoughts,

    Mark

    Reply
  5. kristinawise@googlemail.com'
    My Website says:

    It is definitely popular but have become much more inside last couple
    of years because from the problems inside housing sector My Website it is vital that you realize that they are
    doing cost plenty but for those asking “i want a3000 loan without having credit check” this may are the
    product to examine.

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>