I’ve recently been developing a paper for use inside IBM on the topic of Big Data in the context of Financial Services. I have been working with Big Data technologies in a variety of contexts for the past year or so, and the paper has been a good opportunity not only to explore the topic with my peers, but also to take stock of what I have learned in that time. Whilst the paper is an IBM-specific view, in the process I have been refining my own point of view, and that is what I’ve decided to record here as a series of observations that I’ve made in this time.

Thanks to Mark for his additional review and comments.

What’s in a name?

As technicians we are naturally wont to try and find the absolute meaning of any given piece of terminology, which means that when terms like “Big Data” or “Cloud” come along, a lot of time is spent deciding on what the “true” meaning really is. Published definitions of Big Data vary, generally tend to be at a high level, and reflect the wider strategy of the organisation making it. For example, the IBM web site defines Big Data in the context of the increasingly connected and instrumented world in alignment with the Smarter Planet agenda:

“Everyday, we create 2.5 quintillion bytes of data–so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and from cell phone GPS signals to name a few. This data is big data.”

A cursory look on Wikipedia yields a less applied definition as follows:

“Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.”

I could go on but sufficed to say, trying to tie Big Data down too firmly is clearly not helpful. What is interesting is examining some of the definitions of the term that I have heard myself in a variety of fora, such as:

• Social media analytics.
• Hadoop and MapReduce
• Stream analytics and complex event processing.
• Unstructured data.
• Data gathered from smart energy meters.

It is tempting in such circumstances to critique each example for accuracy and completeness against a chosen definition, but in the end I have reached the conclusion that the answer is that Big Data is all of the above, and many more things besides. This leads me to my first conclusion:

Big Data as a term is deliberately open to interpretation to accommodate a variety of possible lenses through which to view it, and the many and varied definitions reflect this variety.

Noticeable traits of Big Data scenarios

The format and structure of the data are not constrained to those of traditional business data models

One of the key themes of Big Data is the removal of traditional constraints around the type of data that can be leveraged in support of the business. Taking a Hadoop-type environment as an example, a key advantage is that data of any kind can be harnessed quickly from its raw format, without the need for a full scale data modelling exercise.

It is important to position how some of the Big Data technologies fit with the traditional data warehouse approach. One clear difference is the nature of how the data is stored and made available for analysis. Traditional data warehouses store data in well-defined structures to support Online Analytical Processing (OLAP) in the context of business intelligence initiatives. Typically a data warehousing project involves significant analysis to determine the business data structures into which the data is to be loaded for consumption in this way.

In a Big Data scenario, the source data is typically accessed in its raw format e.g. log files, audio, text. There can be a number of reasons for this, ranging from the sheer volume of data that would make traditional handling inefficient and costly to the uncertainty of the requirements and primitive nature of the data which would render a traditional data modelling exercise extremely difficult. Furthermore, the rapidly changing nature of Big Data sources, the business pressures of time to market and agility and the fact that we are only just starting to understand the possibilities also means a traditional approach is unlikely to be effective.

Data may be sourced from a variety of sources inside and outside the enterprise, including the public internet.

Another key point is that from a data ownership perspective, it may not just be about you any more. The “Big” in Big Data may refer to size, but it equally true may refer to scope — i.e. bigger than one organisation alone. It may of course refer simply to sources within an enterprise that have not been put together before, for example analysis of call centre records combined with an existing data warehouse. Social media analytics of the public internet is a good example where data beyond the “four walls” can be integrated with business-as-usual processes to improve performance.

The data itself may be analysed either in static data store or as a continually changing data flow.

As discussed previously, Big Data embraces a multitude of interpretations, one of which is the concept of “Big” indicating speed of data movement, or at least that the underlying data set may otherwise be fluid and/or with a temporal element to the business use case.

Again, the field of social media analytics offers a good example, wherein we are harnessing a constantly varying source of data. This in turn may be coupled with a fluid stream of business queries — for example, measuring the impact of recently-launched or enhanced marketing campaigns. This is a good example of a varying data set where the analysis occurs on a static, point-in-time snapshot of the data — data “at rest”.

In Financial Markets, algorithmic trading is a well known example where”Big” refers to the velocity of change, and the demand for fast response time. In this scenario, the data is analysed “in motion” as a continuous stream, with the Big Data tools providing the capability to spot potentially valuable patterns that indicate particular circumstances are occurring, in this case an order being made automatically at the right time.

Requirements for applications in the environment are often fluid and evolutionary.

As discussed above, to a large degree this is unsurprising given the emerging nature of the subject area. Technology-led exploration grows an increased appreciation of “the art of the possible”, and technologies such as Hadoop are very amenable to agile, rapid experimentation — indeed, one of the key value propositions of Hadoop is the ability to get started quickly and cost effectively, and the agility of the environment.

The ability for technology to handle Big Data in solving business problems removes some of the traditional IT constraints on thinking, and this naturally tends towards an exploratory approach to innovation with analytics. The flexibility inherent in IT tools such as Hadoop enables new degrees of business innovation, potential for value creation, and differentiated products and services. Factor into this the highly competitive and market-driven nature of consumer-facing fields such as retail and consumer finance, and this is a recipe for an ever-changing set of requirements.

“Big” is a subjective measure and specific to the context in question.

“Big” is very much in the eye of the beholder — earlier in this post I talked about the variety of definitions for the term Big Data, and largely this stems from the use of this inherently subjective term. “Big” to a business analyst at a bank may mean too many rows for their standard spreadsheet to handle any more. On the other hand, “Big” to a data-centric organisation Google means something different entirely.

Another definition of “Big” is not as a measure as such, but as an indicator of being “outside of conventional bounds”, for example drawing in data from social media or third-party organisations. In this sense “Big” becomes synonymous with “uncharted” and possibly “hard to manage” within the confines of the traditional enterprise scope.

Having concluded that there are many possible perspectives on Big Data, there is an emerging set of recurring attributes of a Big Data environment when one drops down a level of detail to examine the technical requirements.

Business scenarios for Big Data

It is interesting to note that the terminology itself is inherently technical, which instinctively leads a lot of the current thinking into the world of implementation technology. This naturally leads to a “bottom up” view of the problem space — i.e. here is what particular technology allows you to do, now think how you can apply that capability to your business and see what fits. From a technologist’s perspective, this is exciting because one can see the possibilities, and this natural enables an entrepreneurial approach to IT. This can however end up becoming the archetypal technical solution looking for a problem.

It is interesting to note that there is no one obvious place to start in terms of a business problem space addressed by Big Data. A few are emerging, for example those associated with social media analytics (marketing and campaign management, product development and so on), but actually it is likely that in many cases the Big Data thought is something one goes armed with when the top down analysis and requirements gathering begins, rather than a precise piece part that fits a specific problem. For example, there is not the same defined link as exists between a single view of the customer type business problem and a master data solution.

It is that new art of the possible, and suspending judgement on what can be done that is the real benefit of the Big Data thought from a top-down business perspective.

Whilst there are a growing family of technology pieces in the Big Data solution story, you may not realise you have a Big Data business problem until you get there.

First impressions began with the arrival of the runner’s pack, which contained the chip tag that all entrants wear on the ankle. The event is sponsored by B&Q who are a major local employer and the organisation and quality of the pack contents was also noticeably higher than some others. I was number 1467.

Getting to the run itself was the first challenge – the event is popular and the traffic around Eastleigh was very busy and parking spaces at a premium. Having finally found a space a short distance away, we wandered first round to the race HQ at Fleming Park and then on to the starting line on Passfield Avenue.

The course runs from just outside Fleming Park through Boyatt Wood and up to Allbrook before coming back down Passfield Avenue and through Eastleigh itself down Derby Road and finally back down Chestnut Avenue and back into Fleming Park itself to the finish line. The course is mainly on road, with the local roads being closed for the race and pretty flat, save the hill up to Allbrook.

Whilst I’ve grown in confidence I thought I’d made an honest assessment of my abilities and started probably about a third of the way back. When the gun went off, I realised I may have been a bit hard on myself (or indeed that others had not been hard enough) as it took me a good 500-700 metres to reach my desired running pace. The atmosphere was really good with a nice mixture of running clubs and fun runners (including some guys in Sombreros).

Having got going at around the 4:35-4:40 mins/km mark, the first challenge was the hill up Twyford Road to Allbrook which was enough to drag on the legs but not too hard. As I said before the course is pretty flat, though coming down Woodside Avenue from the peak of the hill the decline was sufficient that I could pick up to 4:15-4:20 pace which helped offset the slow start off the line. Coming back down Passfield Avenue, lots of people were supporting from the sides of the roads which was great.

Heading up Derby Road towards Eastleigh was surprisingly hard, and probably my hardest part of the course. I think it was a combination of a slight incline (though not much) in the road, fatigue from the first half of the race and a very long straight where the next turning was not visible for a while. I noticed a few people flagging so I don’t think I was alone in finding that leg reasonably demanding. Heading back in to Fleming Park and into the funnel I ended up having a final sprint competition with a guy who had crept up on me but he had just a bit more strength left pipping me across the line. Can’t win them all.

Provided refreshments at the finish line included a bottle of water, some (nice) fruit cake and a banana.

Garmin recorded my time as 46:39, average pace 4:40 min/km. The official results recorded my chip time as 46:24 and gun time as 47:08, both of which were a PB over my Winchester time so I was more than happy with that.

The Winchester course is a single loop rather than laps that began at River Park Leisure Centre, and ended at Winchester City FC’s ground just around the corner. The main course heads clockwise north easterly out of Winchester ultimately looping round Kingsworthy and nearby Headbourne Worthy, before heading back in to the finish via Andover Road. This would be the hilliest 10km I had attempted to date, and I was looking forward to the challenge.

On arriving at River Park, it was clear that this event is popular, with runners arriving from all angles. I finally managed to track down my fellow runners Tim and May just ahead of the start. The weather was overcast but dry although the start on the mud and grass was somewhat slippery from the previous days rain. A bit like at Romsey we couldn’t easily hear the briefing and the actual start was a bit hazy but in the end we got away, beginning with a lap of a playing field. We made the mistake of starting a bit too far back in the field, and even by the end of that initial lap people were still sorting themselves out.

Coming off of the grass the course then headed uphill through the residential area of Abbotts Barton nearby and onto the B3047 (Worthy Road) up towards Kingsworthy. The initial hills up to the main road were a taste of what was to come, for although the initial leg into Kingsworthy was by and large downhill, the serious hills began as the course continued through and round towards Headbourne Worthy. In fact the course followed a continual incline between around 4.5 and 8km, all the way through Headbourne Worthy and onto the Andover Road. All the way around the route there were people by the sides of the road cheering us all on which was a great feeling, and there was additional traffic control in place to facilitate crossing would otherwise be pretty busy main roads.

I tried to keep focused on maintaining a constant rhythm – I wasn’t going to wear myself out by trying to outrun the hills, and hoped that I would make up the time on the declines that I hoped would come round the corner. By and large I tried to keep my pace between 4:30 and 4:50 minutes/km on the flat, and 5:00 to 5:15 on the inclines which I pretty much achieved.

After the long slog from Headbourne Worthy, there was a last sting in the tail of a steep incline over the railway as the course headed back in towards Abbotts Barton, and then into the final decline in towards the finish. I was starting to feel tired by this point and decided to maintain my rhythm and let the natural decline of the hill speed me up. As the route led into the football stadium, however, I felt sufficiently good that I decided to go for one last sprint to the finish, and despite the surface changing to a loose shingle, I still managed to move myself up a few places before the finish line.

According to my Garmin, I had completed the race in 47 minutes 40 seconds. The official results listed me as 47 minutes 47 seconds, both of which I was really happy with. My finishing position was 184th out of the 419 men.

I managed to also capture some action shots of Tim and May as they made their way in.

Then of course we had the customary group photo to commemorate another good running outing.

Roll on the Eastleigh 10k in March, this was great fun.

The weather was good – not too sunny as befits the time of year but not raining, and the course pretty flat as it is a looped course mainly on pavement within the grounds of Broadlands. The course consists of three laps, the first two being around 2-2.5km with the third lap having an extension to bring the course up to the full 8km distance.

The start was pretty disorganised and we didn’t get to hear a lot of the briefing, however we did manage to hear the whistle to start and we were off. I was quite pleased as I didn’t feel too nervous, and managed to get into my stride. I managed to get into a pace of around 4:45-4:50 mins/km which I tried to keep up all the way around. Terrain was okay, though one stretch of the main loop was on a looser tarmac surface which had quite a few potholes in and seemed to suck more energy out of my legs than the majority of the route.

On the third lap, the extra length of circuit was a straight length that took you out to a turning point and back the way you came which meant that for a stretch there was two-way traffic. I managed to catch a high five with Tim as he went by on the way back, and I liked the way the members of different running clubs all supported each other as they went by.

Into the final leg towards the finish, I could feel myself tiring but dug in and tried to keep the same pace to the end. There was a guy who I think must have been tracking me on the way round who made a break to overtake at the end but I managed to get a kick in to just level it on the line.

The official results show me as the 142nd man over the finish line (out of 274 men in total) with a time of 38 minutes 37 seconds. I’ve uploaded my Garmin stats for completeness too, will add the link later.

Great day out, roll on the Winchester 10km in February.

Enough is enough

I came to start running again mainly because I’d reached a point where I was so unhappy with the shape I’d gotten into over the past couple of years or so. There were a few reasons for this – having a relatively unpredictable job that involved travelling around and staying in hotels, house moves, a series of weddings and stag weekends and so on. I’d been relying on a weekly game of 5-a-side to try and keep in shape but my working routine typically meant that it was hard to find myself in the right place at the right time on a regular basis. I’d tried a few diets to knock things back into shape but nothing sustained long enough.

My finally deciding enough was enough was triggered by the cumulative effect of a few specific events:

• Going to get a new suit fitted. My existing suit (34” waist trousers let out to 35-36”) no longer comfortably fitted so I was forced to go and get myself refitted. The suits I came away with were very nice, but boasted 38” waist trousers. Now these were fairly generous but nonetheless fitted me, and this made me feel slightly depressed and rather ashamed. At my happiest, I was wearing a 32-34” waist in trousers, 32” in jeans.
• Seeing photos from my friend’s wedding. My wife and I went to a fabulous wedding at St. Paul’s Cathedral in London, followed by a suitably swanky reception in a restaurant nearby. I had not seen my friend for nearly five years since he and his now wife live Down Under, and so was acutely aware (especially wearing the aforementioned new suit) that I did not quite look the same as the rather slimmer version of me that he last saw. A combination of free drinks at the reception and simply being glad for the happy couple meant that it didn’t bother me too much on the day, but when the photos appeared on Facebook, I was that ashamed that I methodically went through and untagged myself from each photo.
• Returning to a hotel that I last visited in 2001. I had cause to stay in the same hotel that I once stayed at back in 2001, when I was feeling significantly fitter and healthier. On arriving at reception I had a flashback from being there before, and of how my first question back then was where the gym was so I could keep my fitness regimen up whilst working away. Nearly ten years on, the contrast between how good I felt then and how I now was pretty stark.
• A colleague talking about his weight loss. On the same business trip, a colleague was telling me about how he had lost a fair amount of weight and was having to buy some new clothes to fit. I remembered being in the same situation in 2001, and how good it felt relative to how I was currently feeling.

So in summary, I was at something of a personal nadir health and fitness wise and felt I needed to do something. I was not in the mood for a quick fix either, I wanted to do something that would both do me good and would also give me a stimulating pursuit and a way of unwinding outside the office. A few years ago I got into the habit of running for enjoyment but had stopped because I had a bad ankle injury. I always enjoyed the running aspect of 5-a-side though, and felt that running would also provide me with the flexibility that it could fit in with my routine and any business travel since you can do it on your own and just need some kit and trainers.

Getting started

A big part of my thinking was how to stay motivated and make sure I kept at it. I adopted a two-pronged strategy: 1) social 2) financial. To accomplish the first, I had a search for an online running community, in the end plumping for dailymile.com. My rationale was that if I was public enough about what I was doing, enough people would get to know (via the Facebook integration, for example) that it would make it hard for me to stop. I am not sure I would have been able to bear telling people that I’d given up. The second part of my strategy was simply to by decent enough kit that in financial terms I would have to make use of it to justify the expense to the household. To this end my new pursuit began with the purchase of some New Balance 850 trainers, expertly fitted by those good people at Just Run in Eastleigh.

I didn’t worry too much about other kit at this stage, since I’d actually feel very self conscious having the proper clothing whilst I still didn’t feel good about myself, and also before I’d got myself going. I opted for my 5-a-side garb of Saints shorts and polo shirt – familiar and baggy enough to hide the multitude of sins underneath.

In terms of a running route, we are fortunate that we actually live on a suitably sized loop of around 3.5km, long enough to be worth doing but not too intimidating, just the sort of thing to get me off and going. And so it came to pass that on Monday 28th June 2010 I took my new trainers out on their first run.

First time out, my expectations weren’t hugely high but I was gutted to find I couldn’t finish the lap, needing to walk for 30 or 40 yards before carrying on. That said, I felt I’d accomplished something by starting and actually not being able to run the whole way around at least gave me something to bite on in terms of progress – I knew I would see progress quickly, and knowing how good that would feel kept me going. I started out with a routine of two days on, one off and I did indeed see results straight away since I went straight round on my second lap.

Two other important milestones came along quickly after in that within a week or so of starting running, I had two business trips that caused me to stay away overnight which would be a good test both of my own application and the flexibility of my routine. The first was a trip to the Cotswolds to the pretty village of Minster Lovell on a course with work. Before heading up I worked out a route, packed my kit and pledged to go on the first night. This I did, on a beautiful summer’s evening, and I felt fantastic afterwards both for the run itself, and for the fact I’d proven I could keep going when travelling with work. The second business trip was to London and I followed the same approach, this time pledging to run first thing in the morning before work on the second day. Again despite a working evening out the previous night, I set my alarm and got up a 6:15am and ran along the South Bank in the early morning sun. As the working day wore on, I felt tired from the late night and early start but had the now familiar fresh feeling of having exercised and enjoyed it.

A further milestone came along a little later, when once again I drew up a route and packed my kit when away for the weekend attending a colleague’s wedding. I ran the morning of the wedding in the Lancashire countryside and once again felt great when I came back to get ready.

Joining the Hursley Parkrunners

As my routine developed, it came to the attention of friends on Facebook who encouraged me to attend the weekly Parkrun 5km event in nearby Eastleigh, as a number of them were regulars. Whilst initially I was not sufficiently confident either in my running ability (I was not timing myself at this stage) or to run in front of others, it gave me the motivation to edge my distance up from my regular 3.5km to a 5km distance such that I could target a Parkrun when I was feeling more confident. To this end, I extended my regular route, and set about practising and building up my comfort with the longer distance.

Again a couple of things happened to finally give me the confidence to sign up. First of all, I had been noticing that clothes were fitting better than they did and I was generally feeling healthier. The second thing was that on leaving the house for a run, I had started to glance at the clock (on the microwave, very scientific) as I left so I had a very rough idea of how long it was taking me to do my new longer route. By this rudimentary approach, and being deliberately conservative, I reckoned on a time of around 27 minutes for a 5km route. Checking out the times on the Parkrun site, it struck me that this would be respectable enough. So, on 21st August 2010, I gave it a go, clocking 25:41 and feeling very pleased indeed.

Having enjoyed my first Parkrun so much, I was hooked, both on the camaraderie but surprisingly on the competitive element, and proceeded to attend regularly. Having some good friends along from work helped a lot and I appreciated their support and encouragement hugely.

I am by no means fast but you find your level, and coming into the home straight I found that on a few weeks I would have a good natured sprint to the end with a fellow runner around my level.

They have a photographer there each week, and this was the first photo that I didn’t feel embarrassed about (a few weeks in) so I’ve included it here for posterity.

Getting some proper gear

One thing taking up running has done is provide a very good theme for birthday and Christmas gift buying. My first piece of “gear” was an armband that I bought for myself to hold the iPod, since I rapidly began to tire of holding the thing in one hand whilst trying to run.

For my birthday, the real kit started to arrive in the form of gifts. No more timing off the microwave, as a Nike+iPod kit duly arrived. Nike+ gave me a much better guide as to my progress and timings.

I’d never realised how much difference having proper running clothing makes. A cotton polo and nylon football shorts soon become uncomfortable. My core running kit began to develop with some Nike running shorts and a Nike Dri-FIT running top.

I’ve since augmented this with additional shorts and tops – I’ve become something of a Nike fan, some pretty much most of what I have is Nike Dri-FIT of some kind. The most recent major purchase I’ve made clothing-wise is a Nike Storm Fly jacket, both breathable and warm for the winter weather.

The difference I felt was amazing, much more than I’d expected. It was also a significant self-confidence hurdle to overcome, since proper running gear is closely fitting and in the case of shorts gripping lycra. The first time I wore all my new garb to Parkrun was a significant milestone for me personally. I’m fully togged out in the picture taken at Parkrun above.

The onset of the recent bad weather (including snow) highlighted how I needed something with more grip to cope with the difficult conditions. I managed a few runs in my standard road trainers (somehow), but finally invested in a pair of Brooks Cascadia trail shoes.

Increasing my distance and developing my pace

I was inspired by friends training for the Great South Run (10 miles) that I set myself the target of improving my distance from the 5km of Parkrun, to something a bit more demanding. I first increased my local run distance from 3.5km to 7km, simply by adding an additional lap to the same circuit, and developing various extensions to add variety. To make things more interesting still, I decided to make a bigger jump and attempt a 10km run, and successfully completed this milestone by running from Owslebury back to home, doing so in around 54 minutes. I was really surprised how natural this distance felt, and encouraged by the time I achieved, especially having not been running for very long. This again increased my confidence further.

Since that point, I was determined not to lose momentum and increased my regular running distance from around the 7km to 10km and then on to 11km. I have one eye on both the Great South Run and a half-marathon before 2011 is out, and so I am determined to keep chipping away at my distance.

One thing I have begun to work on is developing a “long distance” pace as well as a “short distance pace”. As I increased my distance, it became clear that I was hitting a wall blasting around at runs in excess of 10km at my 5km pace. A friend shared with me an anecdote to the effect that most beginners attempt their short runs to slowly and their long runs too fast. I have recently been attempting my longest runs so far (13km) at a pace of around 5:30 per km, with shorter runs (e.g. a Parkrun) at around 4:45-5:00 per km. I’m currently trying to mix longer with shorter as a part of my weekly routine, and gradually increase my longer run to nearer the 17km (10 mile) mark over the coming months.

My goal for 2011 is to regularly have a long run a week of around 17km, thereby comfortable enough with the Great South Run distance, and close enough to half-marathon length.

As of today, my personal best for a 5km Parkrun is 23:38 (in Basingstoke), and 1:09:29 for a 13km long run.

Entering organised runs

At the time of writing this, I have just passed another milestone in that I have entered an organised event in the form of the Romsey 5 mile event which takes place on Sunday 23rd January. I have also now submitted entry forms for the Eastleigh and Winchester 10km events.

All in all

Taking up running has been a revelation these past months. The enjoyment of the exercise and simply feeling fitter and healthier is very rewarding, but also the social aspect of running with friends and the camaraderie and support from others has been hugely inspiring too.

I hope the next six months are as enjoyable.