Micro data integration

This article, written by Neil Mason, was originally published on Clickz.com and is republished here with permission.

There is one thing about the digital marketing world – we are not short of numbers!

Typically we may have numbers coming in from our web analytic systems, our PPC consoles, our affiliate marketing systems, our ad-serving data and so on. On average a marketer may have at least five or six different sources of data, if not more. So, the challenge is to make sense of it all and last time I looked at what I call “macro data integration”. Macro integration is about pulling together data at the summarised level in order to be able to relate different data sets together and spot trends and exceptions.

However, as you want to be able to act far more tactically on your data, you may need to think about “micro data integration”. Micro data integration is where different data sources are integrated at a much more atomic or granular level. Often this is done at the customer level.

Why would you want to think about micro data integration and what are the benefits?

You might want to think about data integration at the micro-level for a number of reasons:

1. To enhance the value of the data that you hold on a customer or a product
2. To enable better diagnostic analysis of marketing activity
3. To be able to execute personalised or event driven marketing programmes

For example, you may want to be able to combine data from a web analytics system on browsing behaviour with online and offline shopping data so that you can be more specific and targeted in your direct marketing activity. Or you may want to look at the long term value of customers brought in by different types of acquisition channel.

Often the question is asked about the best place to integrate the data. Should data be imported into your web analytics tool or should data be exported from the web analytics tool into a CRM system or something similar? The answer is driven by what the objectives are and may involve both activities.

If the objectives are to improve the customer marketing processes then it is likely that the best route will be to export certain data from the web analytics system into the CRM system, as it is usually the CRM system that drives the operation of the outbound marketing activity. The customer database or CRM system provides the total customer view and the data from the web analytics system will be just one component that total customer view along side other data gathered from other systems.

Another reason why you might want to export the data from a web analytics system into another database is because you might want to analyse the data using other tools. Web analytic systems can report data in a variety of ways but there may be occasions when you want to do some more sophisticated statistical analysis using tools such as SAS, Clememtine, SPSS and the like. In some of the work that we do, we process data using a web analytics system to generate visitor level records which we then look at using data mining tools to look for interesting patterns of behaviour.

At other times, it may be useful to enhance the data in a web analytics tool by importing data in from other sources such as the marketing database, customer database or the product database. This is likely to be more useful when you maybe need a site centric view rather than a customer centric view, eg:

• Which type of people look at what type of content?
• Which acquisition channels given the greatest return on investment?
• Which campaigns tend to acquire the least loyal customers?

Last time I discussed the data management challenges around data integration and this is true of micro data integration as well. Thinking about what data to move will require some careful planning. Different data sources have different data structures and they won’t necessarily fit easily together. Often this may mean that the data may need to be manipulated or transformed in some way in order to be able to lay it along side the other data.

The volume of data being exported or imported is an issue as well. This will also impact on how often you do the data integration. Monthly, weekly, daily? We all know that web sites generate huge volumes of data and it is often impractical and unwieldy to extract the data in its rawest format. Think about what you want to do with the data and create summarised variables if possible. For example, if you want to have visit based recency and frequency data in the customer database, then it’s preferable to create a couple of summary variables such as date of last visit and number of total visits, rather than import the whole customer’s visit history.

The good news is that web analytics systems are becoming increasingly open and able to interoperate with other systems. The launch of WebTrends 8 and WebTrends Marketing Warehouse last month is another example of steps in the right direction for making it easier for users to “micro-integrate” their data.

Till next time.