Getting the Most from Your Data Warehouse

Editorial by Bill Bittner, President, BWH Consulting


Many retailers have spent considerable time and effort installing Data Warehouses. When they’re done and begin writing queries against it, it is hard to avoid “getting lost in the trees.” All of a sudden, they have so much detail about each item sold, at what register, at what time, and to whom, that the data loses all meaning. They need a way to extract the “information” from all the data.


This inspires two thoughts.


First, in addition to storing the raw data, the Data Warehouse needs to have benchmarks for comparison. These benchmarks would describe what is “normal” for a particular transaction and serve as the basis for filtering the raw data into exception reports that show where deviations are occurring. These benchmarks will let the user manage by exception. The key is to decide ahead of time what benchmarks are needed, and then have them available in the Data Warehouse during query operations.


The second thought is to approach using the data in the Data Warehouse from a completely different angle. Instead of inquiring against the raw data, first decide what information is needed to drive the various decision processes a retailer goes through. This approach would ask, “What facts are important to take into consideration when making assortment, pricing, promotion, or replenishment decisions?” Then the Data Warehouse would be used to determine these specific facts and they would be loaded into the decision-making applications. The design of the decision-making processes can plan to use these facts when determining what action to take.


Data Warehouse software is designed to answer a lot of “what if” questions typical of a research facility. But retailers don’t ask a thousand different questions. Retailers ask a relatively few questions thousands of times as they evaluate the performance of individual items in various market areas and locations.


Moderator’s Comment: Do you agree that a Data Warehouse can provide too much detail? Can the key “facts” for driving
various decision processes be defined? Are there other ways to get to the “information” that is hidden in the data out of a Data Warehouse?


I really believe the two simple steps I described would improve the return on Data Warehouse investments. Once it is accepted that processes should be driven
by facts extracted from the retailer’s Data Warehouse, it may also be desirable to use facts from other sources such as market research firms. This can provide additional facts
about items the retailer may not even carry.

Bill Bittner – Moderator

BrainTrust

Discussion Questions

Poll

10 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Bernice Hurst
Bernice Hurst
18 years ago

Reading through the comments, several words are flashing across my mind. Greed – just because you can gather data, doesn’t mean you have to. Focus – know what you want to know and then gather the data you need to know it. Don’t try to answer every question that pops into someone’s head. Different departments have different needs, questions and answers. Trying to compile all the data to suit all of them creates those infinitely large forests. It may sound cool to be able to access any little detail that grabs you but databases cannot and should not be all things to all departments. Way back when I was publishing trade directories we had multiple reasons for wanting information on sales, some for the editorial department, some for the advertising department and some for the circulation department. Our weekly reports were inches thick. Reams of paper wasted. Chop down a few of those trees and re-distribute the data. Don’t try to extract the answers to basic questions from complex data.

Gene Hoffman
Gene Hoffman
18 years ago

Answering “what if” questions can be extremely important in retailing as the playing field is constantly being tilted by an ever-revising environment and circumstances. So endless data is collected and stored. But too frequently that data can serve as a security blanket for retailers and a profitable business for the collection-providers. That thought is contestable, of course, but think about it.

It is always thus, impelled by the state of mind which is destined to not to last, that we make our irrevocable decisions of input — at least until tomorrow. So, retailers, watch carefully your cost of both information preservation and your on-going operations.

Don Delzell
Don Delzell
18 years ago

Best practices in data warehouse structure are very complex, and not appropriate for this space. If anyone out there is looking at the issue: invite a consultant. It WILL pay for itself.

I agree with the moderator’s comments in general. Absolutely, the business must address the information it will require to make decisions and implement those decisions. Remember to address this issue across multiple time frames. Information requirements for immediate day-to-day tasks are quite different from weekly or monthly business analysis tasks, which are quite different from seasonal planning tasks.

The front end of a data warehouse is critically important, but it can only provide access to information based on the way it is captured and stored.

Most consulting firms have very good methodology for guiding clients throughout this process. It is NOT intuitive. Very few things provide the ROI they claim. Setting up your data warehouse correctly, anticipating the business needs and requirements…this is one of them.

Herb Sorensen, Ph.D.
Herb Sorensen, Ph.D.
18 years ago

It’s been compared to getting a drink out of a fire hose. From the perspective of mining the data in our own PathTracker® data warehouse, there is a two step process that is helpful. First, find all of the patterns you can and create a catalog of these patterns, with as detailed understanding of WHY each pattern occurs as you can. This is the basis for a set of “rules” that give you a rich understanding of the data. Follow this with an examination of all the stuff that doesn’t fit the rules. These exceptions provide an even richer understanding of what is going on.

The real value comes when you can then predict, if this happens, then that is going to happen. Now you are in the driver’s seat, in a position to say, “let’s do this, so that we can get that to happen.” If “that” is more sales, you’re now in a position to mint money. 🙂

Len Lewis
Len Lewis
18 years ago

You can never have too much information. But you have to know what to ask and to have someone to make sure you are getting usable information from the raw data.

You can have the most efficient data warehouse — hate the word warehousing — but it’s not going to do any good if everyone in your organization just lets the reports pile up on their desks because they don’t know what to do with them.

Ron Margulis
Ron Margulis
18 years ago

Disclosure: RAM Communications is on retainer with Teradata, one of the leading providers of data warehousing solutions.

There can never be too much information, just like someone can’t be too beautiful (although Nicole Kidman comes close). Wal-Mart’s data warehouse is now more than half a petabyte, equivalent to the number of characters in one billion books. Still, they don’t have enough data to ensure they’re in stock on all items all the time.

The key to using a data warehouse effectively is to deploy analytics that make the most of the information being captured. This requires a project team composed of both business users and IT folks, so the reports and alerts created by the system drive optimal processes throughout the organization.

One of the best examples of how a data warehouse can provide insightful and actionable information is the beer and diapers story. The legend is that a number-cruncher examining retail checkout data discovered a strange correlation: a higher-than-expected pairing of beer and diapers in afternoon transactions. Presumably, the data indicated that young fathers were likely to pick up something for themselves as they picked up baby supplies on their way home from work. The story goes on to say that the retailer rearranged its displays to boost sales of both products.

Tom McGoldrick
Tom McGoldrick
18 years ago

I think it was Einstein who said “Information is not knowledge.” Nowhere is this more true than in a data warehouse. Like with any research project it is important to decide what types of decisions will be made based on the data. Then formulate the specific questions needed to retrieve the relevant information, and only then go to the warehouse looking for answers.

Also, habit is a powerful tool. The best data warehouse reporting is often created so that it is easy to read and regularly delivered. Just like inventory levels, margin and other metrics tracked by most businesses it is the trend overtime information that is often the most meaningful. Trends can be your best source for benchmarks.

Bill Bishop
Bill Bishop
18 years ago

Bill makes two great points on how to begin to solve the core issue of how we make cost-effective use of the detailed information from the business to improve business results.

Here are a couple of additional ideas that build on those involving benchmarking and the development of a set of decision processes.

>Recognize that there’s variation in all data sources, which means that it’s necessary to be able to tell the difference between what’s normal variation and a real change in the data. The use of quality control statistics and the statistics that underlie 6 Sigma programs could be of real help to a retailer here.

>Create a way to systematically develop and test hypotheses which focus laser-like attention on specific information and can provide clear answers to questions. This hypothesis process could easily be embedded into the decision processes, that Bill is calling for.

M. Jericho Banks PhD
M. Jericho Banks PhD
18 years ago

As an Account Executive at Ketchum Communications in S.F., one of the most important lessons I learned was the value of ACTIONABLE research. We never, ever presented research results of any kind to clients without an accompanying action plan. It was a rule, you could look it up.

There are two kinds of cube farms: One is the office cubicle setup in many businesses, and the other is a collection of data parameters that are defined by three descriptors – height, width, and length, i.e., a cube. In most cases a data cube involves the three parameters of time, demo/psychographics, and questions asked. By doing one’s homework and precisely defining cube parameters — while avoiding the word “all” — good researchers are miles ahead in providing actionable research. It’s a discipline. This is where good researchers spend several nights working on their rationale. It’s the definition of the question — and the cube — that is most important.

One of the things I most admired about Catalina Marketing during my five years there, and about George Off, my boss, was our definition of research. George said it best: “Why collect a mountain of information and then expect it to inspire questions? Instead, let’s ask the questions first and then collect only the data we need.” Perhaps George knew it at the time — or not — but he defined “data cube” for me and many others.

Brian Yarnell
Brian Yarnell
18 years ago

I would agree with Ron, that there is no such thing as too much information. The data warehouse should be exactly that, a warehouse. It’s up to a human being (with the appropriate tools) to extract the information relevant to his or her burning issue.

In the past, a lot of emphasis has been placed on creating data hierarchies to expedite this process. However, as the amount and variety of data in the warehouse grows, this can be cumbersome. A more feasible alternative is attribute based analysis.

Using OLAP technology and flexible front-end tools such as XP3 from Interactive Edge, it is possible to empower business users to define attributes of products, geographies etc. This flexibility affords business users the ability to ask questions of the warehouse and get answers in terms that they understand.

For example, instead of creating a dashboard that highlights the ‘top X stores’ in a predefined hierarchy (e.g. location, sales team, etc.) business users can segment stores by any number and combination of attributes and ask questions such as “what are the top x stores that are serviced by Sales Team A, have less than 15,000 sq. ft. of floor space and have exceeded the national average for growth over the last 6 months”.

Using attributes and providing the business users with tools that can combine them into flexible segmentation will achieve two goals: provide more value to the business users, and take the burden off of IT to develop specific reporting hierarchies.

Editor’s note: Interactive Edge is a RetailWire sponsor.