The bulk of data warehousing projects fail because they use the wrong database, says Rob Armstrong, the technical director of NCR's Teradata database division.
"When the analysts say 50 to 70 per cent of data warehouses fail, I think that estimate's low - that's just the ones thatadmit failure," he said.
"When you look around, many data warehouses are not providing value; not moving the customers forward."
Key reasons for failure were organisations building data warehouses without having clear business objectives, so they did not understand what return they should be expecting; organisations not understanding the nature of their existing data so they were unable to properly analyse it - "if you have inconsistent data, you will have inconsistent analysis" - and using the wrong tools for the job.
"Most organisations will take a database designed for OLTP (online transactional processing) and try to make it do decision support," Mr Armstrong said in a white paper on the subject.
A data warehouse was not a "thing" but a process.
"To build a data warehouse means creating an environment and process that allows the collection and management of data, and enabling a process by which that data can be transformed into timely information," he said.
Teradata's system, which was revolutionary when introduced in the early 1980s, was parallelism, bringing the processing power of many micro-processors to bear on a query.
Large volumes of data could be handled and asked seemingly impossible questions.
Mr Armstrong said because Teradata systems were expensive in hardware as well as software costs, OLTP database vendors started offering "parallel" versions of their databases. Even now, a Teradata database would start at close to $500,000 and could run into millions.
He said demand for analytical data warehouses was coming from banking and financial service firms, deregulating utilities who wanted to identify their high-value customers, healthcare organisations looking to understand their costs, and retailers wanting to pick up fraud patterns.
A significant New Zealand site was The Warehouse, which used a Teradata system to analyse sales and customer data.
