Unlock the full potential
No longer isolated, the data warehouse creates even greater value by delivering widespread decisioning capabilities.
Organizations are finding it high time they remove their data warehouse assets from solitary confinement in the corporate ivory tower. Leveraging information assets in an active data warehouse1 in support of tactical decision making requires that the business intelligence (BI) solution within an enterprise cooperate more closely with online transaction processing (OLTP) applications to facilitate near real-time data acquisition as well as decision delivery. Enterprise application integration (EAI) plays a crucial role in seamlessly combining decision-making capabilities with traditional bookkeeping systems.
Far too often, the focus of enterprise data warehouse (EDW) construction is on getting data into the relational database management system (RDBMS) repository rather than getting it out. Certainly, acquiring, cleansing and integrating data into the EDW is a critical set of activities. However, the true success of an EDW will be measured by its ability to deliver decisioning capabilities. In the end, a high-value data warehouse is more about exporting decisions throughout the organization than it is about aggregating all of its data into a big bucket.
In the real-time enterprise, an increasing importance is placed on the delivery of operational intelligence in addition to the traditional strategic intelligence typically associated with an EDW. To be successful in this endeavor, data warehouse deployment will need to be more closely integrated into an organization’s enterprise architecture than has been necessary in the past.
A critical component
Deployment of a data warehouse as an isolated information asset accessed with traditional BI query tools will never deliver the full potential of a true enterprise implementation. Full value of the data warehouse is obtained when the breadth of the user community expands to the front lines of an organization in support of operational decision making2. This outcome will demand more aggressive service level delivery; it also depends on an integration strategy between the data warehouse and the enterprise architecture that is far more involved than traditional implementations focused simply on strategic reporting, ad hoc analysis and data mining.
In the real-time enterprise, an increasing importance is placed on the delivery of operational intelligence in addition to the traditional strategic intelligence typically associated with an EDW.
The integration of the data warehouse into the enterprise architecture will require compliance with enterprise requirements. The deployment of decisioning services will necessitate application construction using EAI middleware standards such as J2EE or .NET/WCF for enterprise delivery. From a business perspective, EAI delivers unrestricted sharing of data and business processes among connected applications and data sources in the organization3. To realize this goal, architects must implement a technical infrastructure capable of combining business processes, software and hardware platforms, and standards to allow seamless integration of two or more enterprise systems so that they operate as one (or at least create the illusion of doing so to the business community). A variety of industry buzzwords all focus on this noble goal. Web services, message brokers, application servers and middleware tools all attempt to provide aspects of the framework necessary to realize the EAI vision.
An active data warehouse deployment relies upon EAI for both data acquisition and decision delivery. Proper implementation requires extremely up-to-date data from the transactional processing (bookkeeping) systems within an organization4. Advanced EAI infrastructure can facilitate near real-time data acquisition. It provides a bridge between the worlds of bookkeeping and decision making. When a business event is recorded in the bookkeeping systems, the EAI infrastructure allows the decision-making environment (active data warehouse) to become aware of the event in real time.
The bridge works the other way as well. When analytic applications in a tactical decision support implementation detect the need for action, the EAI infrastructure is used to deliver decisions to the OLTP systems that will be responsible for the associated bookkeeping activities to make each proposed action a reality. EAI with process integration allows for “closed loop” decision making5. Data fed from the bookkeeping environment into the active data warehouse will (selectively) cause event-based triggers to fire (based on business rules) and initiate decisions that are fed back into the operational bookkeeping systems for execution.
Taking action requires cooperation between transactional and decisioning services. In the old days of monolithic architectures, the logic for bookkeeping and operational decision making was hopelessly intertwined. However, in the new world of component-based architecture, service-oriented architecture (SOA) techniques are used to construct smaller, independent (but interoperable) components that use standards-based interfaces to allow integration without the interdependencies imposed by monolithic architectures6.
Services are delivered by component-based “mini” applications and can be mixed and matched to deliver to the unique business requirements of each organization. Closed-loop implementation requires integration of transactional services and decisioning services within a coherent set of business processes. EAI standards as well as emerging standards such as business process execution language (BPEL) provide a framework for the required integration.
Closing the loop may be realized using either synchronous or asynchronous decisioning services. The choice of implementation technique will depend on the nature of the business process. Synchronous decisioning services are required when a business process needs an immediate response to a request for a decision. For example, a decisioning service to assist a call center representative in making the best offer to a customer currently on the telephone will require an immediate (less than two seconds) response from the data warehouse. This must be a synchronous decisioning service with a very aggressive response-time service level agreement. This is an example of next-generation BI focused on operational decision making.
Adoption of enterprise standards within one’s data warehousing practices is vital. Traditional isolation of the data warehouse implementation team needs to end.
Other decisioning services are most effectively deployed using an asynchronous implementation. For example, consider an analytic application that monitors financial transactions to identify opportunities for cross-selling to a banking client. When a large deposit is acquired, analysis determines the most appropriate offer (if any) to provide as a lead to a personal banker for follow-up with the client. This kind of event detection (over-simplified in this description) has been very successful in increasing retention of assets for banks that have successfully integrated such decisioning services with their business processes.
A synchronous implementation of the event-detection decisioning service is clearly not required in the last situation. The lead generation implementations at banks that have deployed this capability involve loading transactions into the data warehouse using traditional batch or streaming data acquisition services. The frequency of data acquisition may be once per day, once per hour or even near real time using a continuous message-based feed. Software event detectives or event triggers are then used asynchronously to identify transactions of interest and determine what (if any) offers are appropriate for the client. In cases where action is required from a personal banker, knowledge propagation occurs from the decision support environment to the lead management system—which is typically implemented as a workflow management system and resides in the OLTP environment.
The role of SOA
SOA provides a framework for implementing modularized decisioning services. Using component-based architecture, decisioning services are created in a way that allows them to be combined and orchestrated uniquely to each organization’s requirements. This is much more flexible than hard-wired implementations using the old-style monolithic approach.
SOA is critical in creating an environment that allows for interoperability of best-of-breed decisioning services. These decisioning services may be acquired as software components from across multiple independent software vendors and even (quite often) as home-grown software. Adhering to Web services standards and using an enterprise service broker architecture allows interoperability across multiple vendor and home-grown solutions. Moreover, the use of Web services and SOA provides interoperability between transactional and decisioning services without the pitfalls of a monolithic implementation. A perfect example of this approach is SAP’s enterprise services architecture (ESA). This architecture allows interoperability between transactional services provided by SAP software and decisioning services provided by standards-compliant, third-party software developers. The middleware infrastructure for ESA is built on top of SAP’s J2EE-compliant NetWeaver product. Microsoft and Oracle are pursuing similar paths to provide interoperability with their respective application suites.
There was a time that data warehouse implementations were relatively isolated within an organization’s IT infrastructure. A new generation of data warehouse architecture is poised to emerge in support of operational BI. EAI enables seamless integration of decision-making capabilities with traditional bookkeeping systems within an enterprise. The specific tools used for EAI deployment will depend on the maturity of the technical infrastructure and sophistication of requirements within the enterprise.
Best practices will leverage SOA implementation of decisioning services to allow interoperability with transactional services and deployment with integration across multiple vendor and home-grown solutions. Adoption of enterprise standards within one’s data warehousing practices is vital. Traditional isolation of the data warehouse implementation team needs to end. Data warehouse architects must engage in constructive dialogue with enterprise architects to ensure compliance with current and future standards within an organization.