Data Integration

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

It was busy week last week for the online marketer in London with the conference season getting into full swing. There was an inaugural Internet Retailing conference followed by two days of Ad-Tech. By all accounts both events were well attended both in the conferences and also on the exhibition floors. This is good news as its showing the industry is thriving and there are more people out there wanting to do more things.

I was at Ad-Tech last Wednesday running a couple of workshop sessions. One of the workshop sessions was entitled “Data integration: Why does it have to be so difficult?” where I moderated a panel made up of representatives from Tangozebra and Logan Todd here in the UK and from Sitebrand giving a North American perspective on the issue.

We didn’t really know what to expect from the audience and we were worried that we would fail to keep them engaged for long on what might be considered somewhat dry and turgid subject matter.

We needn’t have worried. After a cautious start (after all we are talking about Britain here) the audience soon warmed up to the theme which was basically this: “How do I make sense of all these numbers? Especially when I have different tools, supposedly showing measuring the same thing showing me different numbers!”

A show of hands revealed that the vast majority of end users were using at least two web analytics systems to track their online businesses as well as having other data coming in from search engine marketing systems, ad-server reports, affiliate systems and so on. Probably fairly typical of most online businesses today. So, how do you deal with it?

Probably the most consistent piece of advice that came back from the panel was that it’s messy and difficult, so don’t spend loads of time and effort trying to get it all absolutely right. But at the same time be absolutely clear about it is your measuring in the first place and so understand the difference between a click recorded on an advertising system and a visit recorded on a web analytics system. The two are not the same thing.

One of the biggest fallacies in online marketing is that’s totally measurable and accountable. I think we all know that it just isn’t true. Measuring online marketing performance is fuzzy and complicated. For example, how do you attribute sales properly? Typically you may have someone first visiting your site on a search referral, followed by a subsequent visit from an affiliate before converting on a later visit by coming to the site direct. Each of the acquisition marketing channel vendors may have their own rules claiming the sale. How do you avoid paying two or three times over?

The panels’ answer was to set your own rules and decide how you are going to pay for traffic and conversions. But to do this you also need good data, so you need to spend some time and effort getting your tracking systems sorted out so that you fully understand the dynamics of customer acquisition on your site.

As with a lot of things in life there isn’t a “silver bullet” to solving the problems of data integration and data validation. One key aspect that came through from the session was the importance of looking at relative performance rather than absolute performance all the time. We used to have a saying when I was at ACNielsen many years ago that “a trend is a friend”. If the data shows consistent in a certain direction, then it’s probably true. If there is a sudden change, then it’s either as a result of something happening or there’s an issue with the data. Either way, you will want to find out why is happening.

The same principal can be applied to online marketing data. Rather than worry too much about why “System A” is showing a different level of traffic then “System B”, look to make sure that the trends are at least in the same direction. Also, rather than focus to much on is it 10,000 visits or 12,000 visits, look at what’s happening between different visitor segments for example.

With such a data rich environment it’s potentially easy to get consumed by the numbers. The feeling from the session in London last week, that you just need to accept that it’s messy and horrible, get over it and move on. There’s a cost to perfect information that’s probably not worth everyone paying.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

I got a bit of a wake-up call this week. An event which I thought was a long off is getting closer. The Emetrics summit in Washington DC in October is now only a matter of a few weeks away and the event organiser, Jim Sterne, has been marshalling his speakers.

Jim’s developed the approach for this Emetrics summit into a multi-track format. I’ve been given the job of moderating a track on “The Voice of the Customer” and there’s a great line up of speakers including my fellow ClickZ columnist Jason Burby from Zaaz.

I’m really pleased that there’s this track at the conference. If you have looked at any of my other contributions to this column, you will know that I believe that the “web analytics” world for too long has been too “site centric” and not “customer centric” enough. In other words people tend to focus on too much on the analysis of the site, rather than on the people who are trying to use the site.

The inclusion of this track in the Emetrics summit means that we are going to get exposure on some of the issues, challenges and opportunities of working with surveys and other customer data sources alongside the data collected from web analytics systems. I’m looking forward to it.

Jim’s email giving the speakers their instructions for the event made me start thinking about the whole area again and just what some of those challenges and opportunities are. The opportunities are plenty and pretty obvious. Augmenting your understanding of behaviour on the site by adding additional insights into who your visitors or customers are and what they think gives you both sides of the story. I often say to clients that web analytics data tells you what is going on on your website and survey data often tells you why.

However, despite the many benefits many organisations still think of their data in silos. So what are the challenges to getting a more holistic approach to thinking about how the effectiveness of your online marketing programmes?

I think they probably fall into three main areas:

* Technical challenges
* Competency challenges
* Organisational challenges

The technical challenges are around getting the different data sources to sit next to each other in a way that makes it easier to analyse. This is the data integration challenge and I’ve written about macro data integration and micro data integration in previous articles in this column. The web analytics systems vendors are making it easier for us to be able to integrate survey data in with the site data and this is a good trend. Most of the major vendors now say that they can integrate survey data into their systems, but do look closely at exactly what they mean by “integration”. One Scandinavian web analytics company, Instadia, has gone as far as making customer surveys an integral part of their product with the ability to write, launch and analyse surveys from within the system. The survey data that is collected is stored in the same database as the visitor’s behavioural data. That’s what I call integration.

Competency and organisational challenges are probably two sides of the same coin. Analysis and reporting of continuous marketing data and the development and analysis of customer surveys are different skill sets. The web analytics industry is probably not mature enough yet for individuals to have had the opportunity to be exposed to survey work before getting involved in web analytics and vice-versa. Typically these may also be separate functions within an organisation. The web analytics data may be owned by the online marketing function and surveys may be owned by the marketing research or consumer insight function. Each function may not be familiar with the other sort of data and so it’s rare that they are brought together.

So lots of challenges and opportunities but things are definitely moving in the right direction. I’m looking forward to hearing the speakers at the Emetrics summit in Washington talking about how they have met those challenges, I’m sure it will be fascinating. I’d also be interested in hearing from you if you have some good examples of how you have successfully integrated web analytics and customer data.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

There is one thing about the digital marketing world – we are not short of numbers!

Typically we may have numbers coming in from our web analytic systems, our PPC consoles, our affiliate marketing systems, our ad-serving data and so on. On average a marketer may have at least five or six different sources of data, if not more. So, the challenge is to make sense of it all and last time I looked at what I call “macro data integration”. Macro integration is about pulling together data at the summarised level in order to be able to relate different data sets together and spot trends and exceptions.

However, as you want to be able to act far more tactically on your data, you may need to think about “micro data integration”. Micro data integration is where different data sources are integrated at a much more atomic or granular level. Often this is done at the customer level.

Why would you want to think about micro data integration and what are the benefits?

You might want to think about data integration at the micro-level for a number of reasons:

1. To enhance the value of the data that you hold on a customer or a product
2. To enable better diagnostic analysis of marketing activity
3. To be able to execute personalised or event driven marketing programmes

For example, you may want to be able to combine data from a web analytics system on browsing behaviour with online and offline shopping data so that you can be more specific and targeted in your direct marketing activity. Or you may want to look at the long term value of customers brought in by different types of acquisition channel.

Often the question is asked about the best place to integrate the data. Should data be imported into your web analytics tool or should data be exported from the web analytics tool into a CRM system or something similar? The answer is driven by what the objectives are and may involve both activities.

If the objectives are to improve the customer marketing processes then it is likely that the best route will be to export certain data from the web analytics system into the CRM system, as it is usually the CRM system that drives the operation of the outbound marketing activity. The customer database or CRM system provides the total customer view and the data from the web analytics system will be just one component that total customer view along side other data gathered from other systems.

Another reason why you might want to export the data from a web analytics system into another database is because you might want to analyse the data using other tools. Web analytic systems can report data in a variety of ways but there may be occasions when you want to do some more sophisticated statistical analysis using tools such as SAS, Clememtine, SPSS and the like. In some of the work that we do, we process data using a web analytics system to generate visitor level records which we then look at using data mining tools to look for interesting patterns of behaviour.

At other times, it may be useful to enhance the data in a web analytics tool by importing data in from other sources such as the marketing database, customer database or the product database. This is likely to be more useful when you maybe need a site centric view rather than a customer centric view, eg:

• Which type of people look at what type of content?
• Which acquisition channels given the greatest return on investment?
• Which campaigns tend to acquire the least loyal customers?

Last time I discussed the data management challenges around data integration and this is true of micro data integration as well. Thinking about what data to move will require some careful planning. Different data sources have different data structures and they won’t necessarily fit easily together. Often this may mean that the data may need to be manipulated or transformed in some way in order to be able to lay it along side the other data.

The volume of data being exported or imported is an issue as well. This will also impact on how often you do the data integration. Monthly, weekly, daily? We all know that web sites generate huge volumes of data and it is often impractical and unwieldy to extract the data in its rawest format. Think about what you want to do with the data and create summarised variables if possible. For example, if you want to have visit based recency and frequency data in the customer database, then it’s preferable to create a couple of summary variables such as date of last visit and number of total visits, rather than import the whole customer’s visit history.

The good news is that web analytics systems are becoming increasingly open and able to interoperate with other systems. The launch of WebTrends 8 and WebTrends Marketing Warehouse last month is another example of steps in the right direction for making it easier for users to “micro-integrate” their data.

Till next time.

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

During the session on Multi-channel Metrics at Search Engine Strategies in New York a couple of weeks ago, one of the questions that came up was about integrating web analytics data with other forms of data. What was the best way to do this? As is often the case with web analytics, the answer is usually “it depends”.

It depends on what you are trying to do.

One of the challenges faced by the typical digital marketer is the number of different data sources they are dealing with at any one point in time. These might include for example data from:

• Ad-serving systems
• Pay per click bid management systems
• A web analytics system
• Affiliate management systems
• Transactional systems
• Customer management systems

The list goes on. How do you pull all this stuff together to get a sense of what is really going on?

I tend to think about data integration on two levels:

• Macro integration
• Micro integration

Macro integration is where you pull the outputs of the various data systems together in to one place so that you can see relationships between different data sources. Excel is often the ultimate macro data integration tool and legions of analysts all over the country spend hours and hours pulling together spreadsheets from multiple data sources to provide a summarised view of what is going on.

This is not necessarily a bad thing as the process of pulling the data together means that the analysts is looking at it and is likely to spot any data quality problems. However, the problem is that 90% of the analyst’s time is spent in data production and only 10% of it is spent in data analysis. The value equation here is all wrong.

One approach is to try and automate the process as much as possible and get the various data sources into a single database. The data is likely to be processed or summarised, rather than in its original format, but at least it’s in the same place and makes it easier for you to be able to report out across the range of KPIs and key metrics. Inevitably this requires a degree of investment upfront to set up the data feeds correct and the reporting system set up. Standard databases and Business Intelligence reporting tools could be used for this type of work.

At a company that I worked at in the past, we took this approach to reporting on our key metrics. We extracted summarised visit, unique visitor and page view data from our web analytic system and stored it along side other key company data in a datmart designed specifically for that purpose. The extracts were done every night so that the next morning the key site metrics was available and reported against some of our other key transactional data using a Business Intelligence tool (Business Objects in our case).

After the inevitable initial teething problems, the process worked quite well and the questions from management and colleagues changed from asking about when the reports were going to be ready, to what the reports actually telling them in terms of business impact. We were able to go from spending 90% of an analyst’s time on data production, to 90% of an analyst’s time on… Analysis!

Obviously, this doesn’t come without a cost. Managing multiple data sources is a messy business. Significant resources may be required from your IT team and they may or may not have the necessary experience in dealing with this particular type of data. An alternative route may be to outsource the process to companies such as Blackfoot Inc and others who specialise in this type of data integration activity.

Macro integration is the process of pulling the outputs of different data sources together, so that you can more easily and effectively report them side by side. Next time I will be taking a look at the challenges and opportunities in pulling together and integrating data at a much more granular level – the customer. Micro integration as I call it.

Till then…

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

Over the past few weeks I have been taking a look at the variety of data sources available for evaluating e-business performance in addition to the data that comes from your site. These additional sources include audience panels, surveys and focus groups. I’ve also been making the point that purely focussing on web analytics data rarely gives you the full picture.

To talk about this on a practical level let’s take a look at how multiple data sources can be used together to look at a specific business issue such as optimising conversion rates on the site. The simple premise is that if you know who is coming to your site, why they are there and what they are trying to do then you can develop the site to optimise these key customer journeys.

To help digital property owners understand how visitors are interacting with their site we use something called a Customer Journey Framework. This framework is an approach to understanding which visitors are trying to use the site, how they are using it and whether they are being successful in their goals or not. There isn’t a single source of data that will give you the answer to these questions. You need to draw the answers from a suite of different places.

The Customer Journey Framework comprises of three key components:

• Understanding the different types of visitors (Audience segments)
• Understanding why people visit the site (Intentions)
• Understanding usage of the site and the consumption of different content (Content)

Different people come to your site for different reasons and there are bound to be different segments of visitors. The challenge is to work out what the most meaningful segments are for your business that you can use for your marketing and site development activities. This is something we’ll take a look at in the future.

Working out who is coming to your site is where you might use audience panel data, surveys or internal data from customer of registration databases. The reality is that you might need to use all three to build up a true profile of the different types of users that you might have. Audience panels can give you a demographic profile (if your site is large enough) but they may not help you to segment your audience in a meaningful way.

Surveys can help you understand if different types of visitors are coming to your site for different reasons. We call these “intention modes”. What is the visitor’s intention when they arrive on the site? What is their goal? To use an e-commerce example, a visitor may come to a site in one of these modes:

• To browse for something and buy it they find something they like
• To do research for price comparison purposes
• To buy a specific product that they have already researched
• To browse around with no intention of buying anything

Visitors in each of these modes will have different goals and will also exhibit different behaviours on the site.

By linking intentions to visitor segments you may find that some modes are more pronounced in certain groups of visitors. For example, in some work for a high street retailer in the UK we found distinct differences in these modes were evident when we looked at it along age and gender lines. In this particular case, older females were tending to arrive at the site with higher levels of purchase intent than younger females. The younger females were looking to be inspired by the site to make a purchase whereas the older females were more likely to already have in their mind what they wanted to buy.

The final link is then to layer these visitor segments and their modes onto the actual content of the site. This is where web analytics data is important and the linking the behaviours that you see on the site back to what you know about visitors and what they are trying to do. So, in our example above are the younger females looking at different types of products than the older females and so do those products need to be merchandised differently on the site to maximise conversion?

Linking behavioural data and profiling data can be tricky. It’s certainly easier if you can identify at least some of the site’s visitors through say a registration process or a transaction. You can match the profiling data captured in the process with the actual behaviour on the site and use that information to generalise for all traffic. It is also possible to link survey response and site behaviour data as well, though certainly here in Europe you need to be mindful of privacy concerns about identifying individuals.

The framework we’ve looked at here is one example of bringing together data from different sources to get a holistic view of what is happening on the site. It’s also just that – a framework, which can be adapted to suit the circumstances of your own site and information sources.