United Nations Flags



Industry Eye

The Business of Government

Big data technologies promise to smooth the operation of government agencies.

Whether at the federal, state or municipal level, governments must effectively use large data volumes to coordinate programs within and across agencies, share information and use actionable intelligence to influence decisions with more reliable and predictable outcomes. The “business” of government encompasses diverse functions such as social services, taxation, defense and homeland security, which necessitates some variety in platforms and techniques.

Existing systems are typically not able to satisfy the demands of these information-centric activities. But big data technologies hold the promise of finessing those limitations to meet the rapidly expanding need for scalable performance for a variety of mixed-use analyses, such as reporting and discovery across government and third-party data sets.

That’s why the March 2012 White House announcement of its “Big Data Research and Development Initiative” challenged federal decision makers to consider how big data technologies can improve reporting and analytics using large, diverse collections of data. The potential benefits of leveraging public data are huge. An October 2013 report by McKinsey & Company found that open data—machine readable information, particularly government data, that’s publicly available—can help unlock $3 trillion to $5 trillion in economic value annually. Governments in other countries can also realize significant benefits with advanced data technologies.

Business Benefits for Governments

As significant producers and consumers of large—and growing—data sets, government agencies are fertile areas to benefit from big, diverse data:


Continuously monitoring network activities and behaviors can help identify and isolate suspicious activity, including denial of service, exposure or disclosure of private data, website defacement, extraction of intellectual property and other forms of cyber espionage. Cybersecurity addresses the need to simultaneously monitor numerous data streams.

As significant producers and consumers of large—and growing—data sets, government agencies are fertile areas to benefit from big, diverse data.

Fraud, Waste and Abuse Analysis

Big data techniques help agencies transition away from after-the-fact recovery of improper payments and toward proactive identification of fraud before payments are made. An example is the creation of predictive models based on massive amounts of transaction history to identify collaborative methods of fraud, triggering investigations sooner.

Document Management

Large collections of varied documents have accumulated over decades across some government agencies. Big data technologies help collect, collate, index and cross-reference these artifacts to provide searchable access and enable cross-organizational document sharing for lookup and research.

Healthcare Informatics

Pattern analysis, filtering and decision-tree analytics can combine large amounts of documents, images and sensor data with electronic health records to assess comparative effectiveness of medical diagnoses. Ongoing monitoring of population health can be achieved to enhance the overall quality of healthcare and improve outcomes.

Storage Augmentation

Distributed storage on a high-performance, big data platform allows for large data resources to be shared by more users across the agency. Big data systems are increasingly used to augment the storage capacity for analytical systems, such as data marts and data warehouses, to deliver improved sharing and usability.

Integrate Solutions

The availability of open-source tools such as Apache™ Hadoop® that comprise a big data ecosystem has led many organizations to perceive the solutions, which have low initial expenses, as providing a cost-effective way to replace existing data warehouses and analytics systems.

However, it is more reasonable for Hadoop and its commercial variations to complement the proven business intelligence (BI), analytics and discovery methods already used by organizations. Two key factors must be considered before integrating technologies and putting big, diverse data to use:

  • Existing investments are not easily replaced. In the government, like in business, it would be inconceivable to rip out the current production environment and replace it with an emerging technology.
  • Hadoop is just one aspect of a big data strategy. Many high-performance frameworks are already in place in the government and businesses that can satisfy the storage and processing demands for enhanced analytics and predictive modeling. That means different approaches to big, diverse data should be integrated into an overall data warehousing and analytics strategy. This allows organizations to effectively capture, manage and analyze massive amounts of data for applications that support mission-critical activities.

Overcome Common Problems

Increased agility, auditable accountability and precise decision making are driving the use for big, diverse data. Advanced technology can enable governments to improve performance, achieve more accurate results and integrate large data sets. In general, the technology works best for challenges with these characteristics:

  • Large data volumes. Typically, “big” refers to the amount of data that exceeds the existing capabilities for feasible and timely processing.
  • Significant data variety. This includes challenges that can be resolved by extracting meaningful information from data coming from sources with varied structure and content.
  • Limitations on system performance and analytics intensity. This is a barrier when a lot of computational intensity is required, such as for large-scale pattern analyses, complex heuristic algorithms and optimization problems. Performance limitations are also a challenge for applications restricted by the speed of data streaming, access latency or data availability.
  • Lack of parallelization. Business issues that can be broken down into smaller units of work can be executed simultaneously, improving performance through data or task parallelism.

Complement Rather Than Complete

Big data methods and technologies can bring definite benefits to any government organization. Data warehousing leaders such as Teradata are incorporating these technologies into hybrid big data architectures. (See “Integrate Solutions,” above.) At the same time, it would be naïve to presume that the significant investment in an existing system infrastructure and years of application development could be simply replaced with any new technology, let alone by largely unsecure, open-source products running on commodity components.

David Loshin is the president of Knowledge Integrity. He is also a thought leader, analyst, consultant and author.

Your Comment:
Your Rating:

Your article is not new to me because I did 'strategic big data intelligence' since 2008, both for the Government and private sectors. It was challenging enough that they were not willing to accept due to corruption and / or bureaucratic practices, internally as well as externally.

3/17/2014 8:17:56 AM
— Anonymous
Fuzzy Logix