Features
Wave of
the future
TDWI reports on what’s next in the evolution of data warehouse platforms.
by Philip Russom
Alternative data warehouse platforms such as data warehouse and software appliances have emerged and proliferated on a grand scale this decade. So what’s motivating this expanding interest in nextgeneration platforms?
The factors are numerous. First, businesses face uncertainty more often than ever before. Recent history has seen companies repeatedly adjusting to boom-andbust economies, a recession, financial crises, and shifts in global dynamics or competitive pressures. Increasingly, they rely on the data warehouse and related business intelligence (BI) infrastructure to understand change and react appropriately.
Also, platforms must be updated to support evolving business requirements. In fact, many technologies associated with the next-generation data warehouse relate to change in some way—such as advanced analytics, scalable architectures, virtualization methods, reusable services and real-time integration with operational applications. Additionally, successful data warehouses mature through multiple life cycle stages. This usually invokes adjustments to the underlying platform and elsewhere in the BI infrastructure.
Successful data warehouses mature through multiple life cycle stages. This usually invokes adjustments to the underlying platform.
Vendors offer more choices
Recent interest in columnar databases has led to several new vendor products and renewed enthusiasm in older ones. Open-source Linux is now commonplace, and open-source databases, data integration tools and reporting platforms have established a firm foothold. In the hardware realm, 64-bit computing has enabled larger in-memory data caches, and more vendors now offer massively parallel processing (MPP) architectures. And leading database vendors have added capabilities and products conducive to data warehousing.
In addition to features within the data warehouse platform—especially its database—a growing number of practices are demanding support by the platform. These include real-time integration with operational applications, various types of advanced analytics, and reusable interfaces exposed through Web services or service-oriented architecture (SOA). Furthermore, BI platforms are now readily available through software as a service (SaaS) and cloud computing.
Click to enlarge
With so many options, it can be difficult for data warehouse professionals and their business sponsors to keep track of technological advances and select the ones appropriate for their needs.
To help organizations understand the alternatives available to them, The Data Warehouse Institute (TDWI) surveyed 417 corporate IT professionals, consultants and business sponsors/users in May 2009. “The Next Generation Data Warehouse Platforms” report catalogs the new products, features and techniques that have appeared this decade, as well as notable advances in more established data warehouse platforms. While its focus is on technology, the report also explains how adoption of cutting-edge data warehouse platforms is driven by real-world business and organizational needs and requirements.
Retain or Replace
Two primary paths can be taken to reach the next generation of your organization’s data warehouse platform:
- Retain the current platform but do more with it. With many deployed platforms, users haven’t tapped all of the capabilities. These users frequently speak of early to mid-life project phases during which it was time (defined by business readiness) to embrace the platform’s more sophisticated capabilities, especially real-time functions, data federation, in-memory processing and analytics. Rather than starting from scratch, a common approach is to remodel the data warehouse significantly to add value without replacing the platform that manages it.
Also, some cutting-edge platform implementations involve tools that are tangential, such as solutions for data integration, quality, master data and reporting. Incremental additions to hardware are common, such as installing more CPUs, memory or storage. These satisfy next-generation requirements—like fast queries, in-memory databases and scalability—by doing more with the current platform.
- Replace it and build out the new one. Ripping and replacing a data warehouse platform is expensive for IT budgets and intrusive for business users; therefore, this path generally should be avoided. Yet almost half of respondents surveyed say they are contemplating this approach. Forty-six percent say they anticipate replacing their current primary data warehouse platform by 2012. Nearly half have no plans to replace their current platform.
Click to enlarge
Move forward
So what is motivating organizations to replace their current primary data warehouse platforms? In the TDWI study, five technology drivers emerged that lead to a next-generation data warehouse. All five are IT’s reaction to urgent business goals:
- Analytics of various types help the business cope with change and discover opportunities. Many data warehouses have evolved to support reporting or basic online analytic processing. As users try to move beyond these basic efforts, 40% find their platform “can’t support advanced analytics.” Meanwhile, the use of advanced analytics is growing because of enterprises’ need to understand constantly changing business environments, as well as to discover opportunities for cost reductions and new sales targets.
- Real-time and related technologies are enablers of operational excellence. Because of economic, competitive and quality issues, many companies are under pressure to achieve unprecedented levels of excellence. Thus, they are embedding data and functionality available from their BI and data warehouse infrastructure into their operational and transactionalapplications. This facilitates time-sensitive business practices, such as operational BI, on-demand dashboards and performance management, and just-in-time inventory and manufacturing. Technology people talk about integrating BI and operational systems through real-time data warehousing. But these are just enabling technologies that help satisfy more fundamental business requirements for operational excellence.
- Scalability, in many senses, powers business growth. A platform can’t cope with growth over time if it “can’t scale to large data volumes” (37%) or it “can’t support a large concurrent user count” (20%). As organizations grow, automate more business processes with software and depend more on BI and data warehousing, they generate more data that needs processing for BI and to run the operation. Likewise, there’s growth in user communities, reports, analyses and so on. With the enterprise data warehouse at the heart of BI—and more and more at the heart of operational excellence—scalability has become a critical success factor.
- Addressable memory space automates new time-sensitive business practices. In particular, various types of in-memory databases can now be far larger than before, and data operations in memory are far faster than those that involve I/O with disks. A leading reason for replacing old server hardware is that the “current platform is 32-bit, and we need 64-bit” (15%). The primary advantage of 64-bit hardware and software is its large addressable memory space. Imagine putting an entire data mart or data warehouse in memory. Reporting and analysis functions are now so fast that they can be easily embedded in operational applications. This speedy intelligence takes business practices to a new level, such as up-sell/cross-sell guidance in telemarketing, automated recommendations in e-commerce, improved service in call center and similar online applications, and fraud detection.
- Architecture and related practices affect a data warehouse’s ability to support a business. Most technology and business drivers involve creative adjustments to the logical architecture of the data warehouse’s data model and the systems architecture of its hardware configurations. When moving to a next-generation data warehouse platform, expect to make architectural adjustments to accommodate new technical functions and their related business requirements.
Click to enlarge
Look ahead
A new-generation platform is probably in your near future. While almost half of respondents are planning a data warehouse platform replacement by 2012, many others anticipate keeping their current platforms but updating them significantly.
Regardless of which path you take, what’s next for your data warehouse platform can vary tremendously. It might tap into leading-edge features like appliances, open source and cloud computing. Or it might simply get you caught up with somewhat more established practices for real-time, advanced analytics and services. Sometimes, a cutting-edge platform addresses administrative issues, such as hardware upgrades, data migrations or architectural adjustments. So keep in mind that a next-generation platform is a relative concept. It depends on where you’re starting, what requirements you must address and how many resources you have.
Editor’s note: This article is based on a TDWI report, titled “Next Generation Data Warehouse Platforms,” by Philip Russom.
Philip Russom is the senior manager of TDWI Research at The Data Warehousing Institute, where he oversees research-oriented publications, services and events.