New Data Management Technology Helps Narrow the Data Gap
| Data Gap Awareness on the Rise |
|
Seisint, in conjunction with ORC International, conducted a survey of 605 management team representatives including C-level executives, presidents and senior business executives. Those surveyed were asked about their companies' data management and recognized challenges barring the effective use of that data.
Identifying the Problem
Not surprisingly, respondents on average say that only 66% of the data that would prove useful to their companies' decision-making process is accessible.
Nearly half (48%) of the executives recognize that their companies currently gather and house more data than they can effectively use. This is what Seisint refers to as the Data GapSM. Additionally, approximately 70% of the executives also agree that their company needs information faster than their people can get it (see Figure 1).
Acknowledging the Barriers
Overall, the executives polled agree that the two largest barriers to effectively using all the data are (1) the inability to pull information from disparate databases and (2) the time-intensiveness of processing that data.
Separately, CIOs and CTOs identified two other issues (1) the overwhelming amount of data received, and (2) the current technology's limitations to manipulate it.
For a copy of the research, please contact Seisint.
|
Enterprises are now doubling the amount of data they collect every year, and the data is getting increasingly more granular and complex. To wrestle this unruly amount of data into an asset that generates value calls for a radical new approach.
Earlier technology revolutions were sparked by the transistor, the PC, and the Internet. Each of these developments required a paradigm shift in business thinking and approach to remain competitive. As with any revolutionary development, the baseline was reset to zero; even industry leaders had to jockey for position on the starting line. Organizations that read the trends wrong or stumbled at the gate often never recovered - remember Wang, Novell, and DEC?
Now, new disruptive technology is required to conquer data. There is an enormous gap between the vast reservoir of raw data collected and an organization's ability to leverage that data to improve decision making in optimizing operations and customer relationships.
The Data Gap
In the rush to gather as much information as possible on operations and customers, including their purchasing history and buying patterns, organizations are amassing unprecedented volumes of data. Unfortunately, neither current analytical technology nor business processes support timely, accurate transformation of this data into business intelligence and action.
The gulf between data collected and readily accessible information is widening. This is the "Data Gap" - and it poses a serious challenge to all growing enterprises. If knowledge is power in the information economy then most organizations are running with their batteries drained. The majority are not able to effectively and efficiently tap into the veins of golden information trapped in their data mines.
Inaccessible information equates to forfeited profits. Yet the Data Gap continues to widen. According to a recent study conducted by the University of California at Berkeley, more data will be created in the next three years alone than in the previous 300,000 years together. Seisint's own analysis of current research by Dataquest and Gartner indicates the amount of digitized data worldwide nearly doubles every year. In addition, American companies currently spend more than $80 billion on technology and services per year to narrow the Data Gap(1).
Business Competition Drives Data Growth
To remain competitive, businesses must develop a deep understanding of their customers. Once an organization is able to decipher the detailed buying patterns and demographics of its customers, it is able to launch sales and marketing efforts into narrowly defined market segments.
Companies require ever-larger amounts of data to segment customers into these ever-thinner slices. The goal is micro-marketing - getting the right products to the right people at the right time. This desire to know customers well enough to accurately target markets is the chief reason that enterprise data is currently overflowing the corporate infrastructure. Today, point-of-sale (POS) retail systems collect every piece of available data about customers. Computers store it. Sales and marketing divisions demand access to it.
Operational demands fuel the growth of these data banks as well. For example, immediate access and verification of credit information or shipping addresses can streamline business processes, provide cost savings, and deliver a competitive advantage.
According to a report by Survey.com(2), a market research firm in Campbell, CA:
o Organizations worldwide are expected to increase data warehouse spending from $37.4 billion in 1999 to $148.5 billion by 2003 - or 43% each year.
o Average volumes of data usable for warehousing are forecast to balloon from 393 GB in 1999 to 1.1 terabytes by 2003.
o The average number of data warehouse users is anticipated to expand from 626 per organization in 1999 to 2,718 per organization by 2003.
Figure 1 - Data awareness on the Rise (6)
Key drivers in this explosive growth of data warehousing are: continued adoption of customer relationship management systems, particularly customer profile analysis; rapid growth in analyzing enterprise resource planning data in a warehouse; and Web-enablement of data for use in business-to-business e-commerce.(3)
Data flows into enterprises from all directions: points of sale; store-level marketing studies (that can rival the United States Census in their detail and breadth); syndicated information; information from suppliers, industry associations and private research firms. In fact, the retail sector has reached the point where the sale itself may no longer be the most important part of the transaction. The atmosphere is so competitive; the most valuable component is sometimes the demographic data collected - data that will improve the company's chances of selling to that customer again. And it is not just customer data spinning in the corporate drives. Companies also collect data about the marketplace and their competitors. As companies go global or launch e-commerce initiatives to expand their market reach, their need for data also expands.
Figure 2 - Worlwide Data Storage Compounded Annual Growth Rate [CAGR(4)]
Technology itself has played a major role in the growth of data volume. As the price of technology decreases, more data is collected and stored. In addition, an ever growing population of users demand access to more data.
To illustrate the growth of data required to support just one evolving application, consider the history of the retail space. Before the development and institutionalization of the bar code, retailers were privy to extremely limited information with regard to the tracking of their merchandise and their customers' preferences. Most, if not all, transactions were conducted with cash, and retailers were for the most part restricted to undifferentiated information on their gross receipts. With the introduction of the bar code, however, retailers gained access to much finer data on the movement of individual products, and for the first time were given the ability to identify trends - which items were being sold most quickly, when and where.
By far, though, the most revolutionary development for retailers has been the Internet, which has opened the doors to a world of clickstream data. Not only can retailers now identify which of their customers is buying what, they can also tell with great precision what steps each individual customer took to arrive at their website, as well as the way in which each customer chose to navigate the site once he or she was there. A vast amount of data is now available not only on their products and their customers' preferences, but also on the entirety of the overall shopping experience. Clearly, each step in this revolution has required dramatically larger amounts of data to support the application.
Compounding Complexity
While it is clear the volume of data is growing at an alarming pace, the more critical issue driving the Data Gap phenomenon is the growing complexity of the data. Organizations need to establish relationships between individual components of the data for it to be useful. For each item or data point, there may be hundreds of unique attributes that can be related to it. This further stresses a company's ability to refine the data into useful intelligence. For example, retailers can gather information about each individual sale. For each item they can track: where, when and for what price it sold; what the color and size was; and what other items were purchased at the same time. Effective analysis can be complicated by the interaction among the attributes. To properly analyze seasonal buying trends, for example, the effect of reduced prices in some locations must be considered.
Figure 3 - Worldwide Complexity Growth
Figure 4 - Growth vs. System Performance
As each new variable is introduced to the database, the corresponding complexity is not incremented by one, but by a factorial of the current complexity. The load on systems processing the data into information becomes burdensome. But as companies are successful and scale up their operations, the data stored mounts and its complexity grows exponentially.
Seisint's research has indicated that, as a rule of thumb, growth in data complexity is equal to or greater than the growth of the data squared [i.e., ComplexityGrowth = DataGrowth(2)]. For example, a retailer might conservatively add an extra 1,000 product SKUs to their existing 20,000 SKUs over the year and go from tracking 20 attributes to 21. The number of products carried and attributes has thus been increased by 5%, but the data complexity has been increased by 13%. If the retailer also wants to add a new level of information to their products, such as inventory by region, or more detailed attributes related to each SKU, the complexity skyrockets.
While Dataquest(4) estimated worldwide data will grow by a compound annual rate of nearly 80% between 1999 and 2003, Seisint estimates that the corresponding complexity is growing at over 200%. This complexity is driven by data relationships and the desire of companies to understand and leverage these relationships to achieve improved business performance.
Tech-Lag
For the most part, data management technologies that are in use today were invented over a decade ago before the Internet became the phenomenon it is, before business ran at the speed of light. The ubiquitous use of productivity tools such as email and cellular phones has created an expectation of instant access to both people and information - and instant response.
Seisint believes new systems and applications specifically designed to solve today's business problems are required. One such approach applies massively parallel technologies to access, analyze, and manipulate billions of records per second (BORPSSM) - even if those records reside in disparate databases. New technologies should also be scalable and flexible enough to work on multiple computing platforms, over the Internet or within private networks, and on distributed or centralized computer systems. Enterprises need to be able to expand from a few gigabytes to many terabytes without negative impact on performance.
New data management technologies are required to enable rapid, efficient transformation of data into useful information. Companies must take action to avoid being vanquished by the Data Gap.
Conquering the Data Gap
Conquering the Data Gap will mean achieving breakthrough business performance. Organizations must be able to perform data manipulations in minutes or even seconds that now require days or weeks. As companies conquer the Data Gap, the competitive playing field will be dramatically disrupted.
In the next several years, Seisint predicts the growth of data will overwhelm businesses that do not find a way to meet the challenge of the Data Gap. Organizations must implement systems and technologies that can support the need to rapidly differentiate good data from bad or useless data, and to effectively adapt business processes that are currently overloaded by data.
Seconds matter. People want instant information. Businesses have to be able to streamline operational processes and automatically verify data. Whether customer contact is through point-of-sales, call centers, sales branches, or the Internet, organizations will have only seconds to convince a customer to buy. Companies need to have a complete, accurate, fact-based understanding of their customers and prospects at the moment they deal with them. Few companies do.
Leaping Ahead of the Competition
Gartner predicts at least 50% of Financial Services Providers (FSP) will have poor customer data quality through 2004 (0.8 probability). For 80% of large FSPs, inadequate data quality will result in the failure to achieve the stated goals of one or more Customer Information Management and Application initiatives per year (0.7 probability) during this same period.(5)
Leading FSPs are addressing this issue. Using new technology, Equifax, a global $4 billion information management company, will introduce rapid code generation and improve computer processing speeds 100 to 300%, providing program value not currently available to customers. Equifax will collapse the turnaround time needed today for credit marketing services fulfillment from an average of 15 days to less than a week. In some cases the time will be reduced to hours.
Another organization that has conquered the Data Gap is Naviant Marketing Solutions, a Newtown Square, PA, company that delivers precision marketing solutions to Web advertisers, publishers and consumer marketers. Its flagship product is its proprietary High-Tech Household File database. This database contains detailed information on more than 36 million Web-connected households and continues to grow. Naviant was finding it extremely difficult to extract accurate information from its database. Complex query responses took days and even weeks to complete, and were executed against a limited number of 20 attributes.
New data management technology helped Naviant use their High-Tech Household File in formerly impossible ways. With unparalleled speed, flexibility and consistent performance, the new system now allows Naviant to rapidly deliver real-time fulfillment of its customers' requirements for utilizing any combination of the hundreds of attributes contained in its database.
It now takes only seconds for Naviant to obtain answers to complex queries. Because the technology supports the ability to append additional data from other databases to the High-Tech Household File, Naviant can now search using 250 important characteristics instead of just 20. Because it is highly scalable, the technology has been able to keep pace as Naviant's High-Tech Household File grew from 10 to 36 million records, with no loss of speed or performance. In addition, Naviant's list fulfillment was accelerated by 300%, which translated directly to bottom line benefits.
Conclusion
As enterprises seek to improve their competitive position through the use of customer relationship management systems and ever-narrower targeted market segments, their hunger for marketing support information leads to massive increases in the volume of data collected and used. Unfortunately, their ability to gather the data far outstrips their capacity to derive timely, relevant business intelligence from these data fields. Seisint calls this capacity differential the Data Gap.
Compounding the Data Gap is the exponentially increasing complexity of the data, which puts a further strain on the ability of existing technology to refine the data into useful information.
The arrival of the Internet and the ubiquitous use of productivity tools such as e-mail and cellular phones have created an expectation of instant access to both people and information - and instant response. Business must run faster than current technology can support. New technologies are required to successfully address today's business problems and enable rapid, efficient transformation of data into intelligence.
Conquering the Data Gap means achieving breakthrough business performance. Organizations must be able to perform data manipulations in minutes or even seconds that now require days or weeks. As companies start to conquer the Data Gap, expect the competitive playing field to be dramatically disrupted.
References
1 Seisint, Inc.
2 1999 Database Solutions III Report, North America - as reported by Mark Hammond. "Survey Traces Huge Growth in Data Warehouse Market." eWeek 13 Dec. 1999.
3 Ibid.
4 1999 Server Disk Storage and RAID Worldwide Market Statistics and Forecast, Dataquest Inc.
5 Knox, Mary Gartner Strategy/Trends, "Cost of Missing, Inaccurate and Inconsistent Customer Data." 5 Sept. 2000.
6 Seisint, Inc. - commissioned study "Data Gap Survey," conducted by ORC International. 22 May 2001.
