Trends in Big Data requirements

Big Data is still emerging and maturing as a style of solution for particular types of problems. The current challenge for both the IT industry and business leaders is to try and make sense of what opportunity Big Data thinking and related technology really creates in an applied sense. It may be that in fact one day we will simply drop the “Big” prefix – today’s “Big” data will naturally mature into augmentations of standard information management architectures. For today, however, as with all new things we are still learning about the possibilities.

Common patterns for Big Data

Even at this early stage on the Big Data journey, we have discovered some specific use cases. In the IBM ebook “Understanding Big Data”, the authors describe six recurring patterns or fruitful areas for Big Data that they have identified during client engagements:

  1. IT for IT log analytics.
  2. Fraud detection.
  3. Social media analytics.
  4. Call centre interaction analytics.
  5. Financial risk modelling and management.
  6. Big data and the energy sector – analytics of sensor data.

These reflect the collective experiences with Big Data thinking and technology to date, and it started me thinking about how that list could grow with new scenarios  aligned to business outcomes that will resonate within a variety of industries.

Take an example of a bank that is trying to attract new customers from a particular demographic to a premium product with various incentives. They want to select the right incentives to maximise the return on their investment in the new product, gain market share from competitors and attract “good” customers (and so on). None of that business intent contains the words “Big” or “Data” yet we know from our early experience that social media analytics has a role to play in terms of better understanding the target audience and, importantly, the competition during product development. So how did we get there?

From use cases to business themes

There will clearly be many more such scenarios that we have not yet unearthed, and so this has caused me to consider whether underlying the known set of patterns that we understand today there is a set of business themes that will help us identify future use cases for a Big Data style of solution. In taking a step back, we might hopefully become better equipped to take many steps forward into the specifics once again.

In order to test this theory, I’ve identified five such themes based on my own experiences with Big Data in the field to date and insight gathered from colleagues and various papers and lectures on the subject. They are as follows:

  1. Augmenting a partial view of an entity or process.
  2. Understanding people better.
  3. Improving management information.
  4. Increasing confidence in decision making.
  5. Supporting partnership and value creation.

The first thing I will note is that there is natural overlap between some (or indeed possibly all) of the above when listed together. Once taken to a suitably high level, the lines between any group of related concepts naturally blur. However the intent is that depending on the mindset and perspective over the business problem at hand, one may well recognise one (or some) more strongly than others. Having done so, one may hence consider that Big Data may have a role to play within a technology solution. This is based on personal perspective, so there may well be other themes I’ve not yet identified.

A short summary of each of the themes I have identified follows.

Augmenting a partial view of an entity or process

This theme speaks to the notion of “Big” as meaning that the underlying data is gathered from a broader variety of sources than the traditional enterprise data warehouse or other data sources within the firewall of an organisation. It is often the case that the success of a particular business process has critical dependencies on external factors outside the direct control of an organisation – for example the weather.

Whilst of course we cannot directly influence something like the weather, if we can analyse its relationship to understand, say, how it affects the performance of our logistics processes against service levels, we can better tune the elements of the process we do control based on that insight. This also speaks to the financial risk modelling pattern mentioned earlier. If we can glean any further insight from external sources as to the position of the counterparties upon which we are dependent, say, we are far better informed to manage our risk position effectively.

Understanding people better

Whatever the core business of the organisation, it is highly likely that at some point meeting a particular business challenge requires a better understanding of people. Possible scenarios might range from a deeper understanding of customer preferences and needs, to understanding the morale of the workforce. Human beings are of course not digital entities and as such operate in an inherently unstructured, unpredictable and fluid manner, whether that is in written text, spoken word or implicitly via their actions.

We can try and impose a structured approach such as a survey or questionnaire, but that is a model that is inherently limited in its breadth and also its ability to capture the finer nuances of opinions implicit in behaviour or the spoken word. By gathering a large volume of data from a variety of sources, be that social media, call centre logs, explicit surveys and the digital footprints of individuals (e.g. entering and leaving a building), we are likely to build a much more accurate picture. Furthermore, we start to build an implicit picture rather than one aligned to the set of explicit questions or pathways we may have led them to.

Improving management information

Closely related to the first theme of an augmented view of a key entity, it is often a reality that an organisation often lacks the level of basic information from its core systems that it would ideally desire to run the business effectively. In seeking to address this issue, we discover that the supporting IT systems were not designed to support the reporting required, or indeed are constructed from a variety of technology that renders the solution complex and costly to modify (or replace) to meet the business need.

Whilst the formal metrics may not be explicitly codified into the solution, a Big Data approach views the vast quantities of “digital exhaust” typically generated by the IT systems as a valuable source to be harvested. By harvesting this output, we can begin to deduce certain of the key performance indicators required in a more cost effective fashion. Taking an approach that uses Big Data principles offers at least an alternative to a long and costly integration or replacement exercise, and has the potential to offer more benefits more quickly. It is important to note also that this theme applies both to applications supporting the line of business, and also the business of IT within the organisation. For example, harvesting server logs in conjunction with support ticket data and call records could yield valuable insights into driving operational efficiency within IT support functions.

Increasing confidence in decision making

Rather than decision making in general (which it could be argued all analytics or business intelligence supports), this theme refers to specific, fine grained business decisions such as whether to extend a line of credit, whether a loan application might be fraudulent or indeed where to allocate stock in a retail chain. Today such decisions are supported by IT systems that are fuelled by large quantities of structured data gathered from a discrete set of sources closely related to the business.

This theme, therefore, derives from the recognition that in addition to these traditional, structured data sources, confidence can be further increased by assessing a broader variety of inputs. For example, mixing social media data with traditional forecasting and inventory data in retail could provide invaluable early insight into coming retail trends in regions ahead of the demand. This could be the difference between sales won and sales (and customers) lost to competitors. Similarly, building a richer picture of an individual (or demographic) or an organisation can only lead to a refined decision making process when deciding whether to issue credit or check for fraudulent activity.

Supporting partnership and value creation

An alliance between two organisations leads to a spectrum of possibility in terms of business model innovation, and also from an IT perspective necessarily has a multiplier effect on the data already available and subsequently created. In this context, a Big Data approach can add considerable value in terms of realising benefit from this increased variety of data, both in terms of the increased variety of data consumed and created, and also the inherent flexibility and speed to value elements of Big Data technology.

Firstly, the data itself may have provided the original impetus for the alliance – each organisation holds pieces of the jigsaw and by bringing the pieces together, they both realise shared advantage. For example a bank and a retail chain may decide to collaborate with their focus on driving increased revenues through richer customer analytics. Big Data thinking in this context provides the thought processes and technology tools to help realise that innovation quickly and cost effectively. Secondly, having developed a shared offering, the resulting service will generate a “digital exhaust” and bi-products quite unlike anything either party could have produced themselves.

In summary

We are at the beginning of the Big Data journey, and one of the most exciting aspects is that we are still scratching the surface of what might be possible if the current pace of technology evolution continues. The above list will doubtless look different in five months time, let alone five years and is in no way meant to be exhaustive, but hopefully the approach will help identify further opportunities for Big Data to drive the business agenda forwards, and develop our set of applicable use cases further.


3 responses to “Trends in Big Data requirements

  1. Pingback: Trends in Big Data requirements | Martin Gale's blog

  2. Pingback: Trends in Big Data requirements | felicevitulano

  3. Nice article, big data is going to play a key role in data and predictive analysis.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s