The Business of Government
Big data technologies promise to smooth the operation of government agencies.
Whether at the federal, state or municipal level, governments must effectively use large data volumes to coordinate programs within and across agencies, share information and use actionable intelligence to influence decisions with more reliable and predictable outcomes. The “business” of government encompasses diverse functions such as social services, taxation, defense and homeland security, which necessitates some variety in platforms and techniques.
Existing systems are typically not able to satisfy the demands of these information-centric activities. But big data technologies hold the promise of finessing those limitations to meet the rapidly expanding need for scalable performance for a variety of mixed-use analyses, such as reporting and discovery across government and third-party data sets.
That’s why the March 2012 White House announcement of its “Big Data Research and Development Initiative” challenged federal decision makers to consider how big data technologies can improve reporting and analytics using large, diverse collections of data. The potential benefits of leveraging public data are huge. An October 2013 report by McKinsey & Company found that open data—machine readable information, particularly government data, that’s publicly available—can help unlock $3 trillion to $5 trillion in economic value annually. Governments in other countries can also realize significant benefits with advanced data technologies.
Business Benefits for Governments
As significant producers and consumers of large—and growing—data sets, government agencies are fertile areas to benefit from big, diverse data:
Continuously monitoring network activities and behaviors can help identify and isolate suspicious activity, including denial of service, exposure or disclosure of private data, website defacement, extraction of intellectual property and other forms of cyber espionage. Cybersecurity addresses the need to simultaneously monitor numerous data streams.
As significant producers and consumers of large—and growing—data sets, government agencies are fertile areas to benefit from big, diverse data.
Fraud, Waste and Abuse Analysis
Big data techniques help agencies transition away from after-the-fact recovery of improper payments and toward proactive identification of fraud before payments are made. An example is the creation of predictive models based on massive amounts of transaction history to identify collaborative methods of fraud, triggering investigations sooner.
Large collections of varied documents have accumulated over decades across some government agencies. Big data technologies help collect, collate, index and cross-reference these artifacts to provide searchable access and enable cross-organizational document sharing for lookup and research.
Pattern analysis, filtering and decision-tree analytics can combine large amounts of documents, images and sensor data with electronic health records to assess comparative effectiveness of medical diagnoses. Ongoing monitoring of population health can be achieved to enhance the overall quality of healthcare and improve outcomes.
Distributed storage on a high-performance, big data platform allows for large data resources to be shared by more users across the agency. Big data systems are increasingly used to augment the storage capacity for analytical systems, such as data marts and data warehouses, to deliver improved sharing and usability.
Overcome Common Problems
Increased agility, auditable accountability and precise decision making are driving the use for big, diverse data. Advanced technology can enable governments to improve performance, achieve more accurate results and integrate large data sets. In general, the technology works best for challenges with these characteristics:
- Large data volumes. Typically, “big” refers to the amount of data that exceeds the existing capabilities for feasible and timely processing.
- Significant data variety. This includes challenges that can be resolved by extracting meaningful information from data coming from sources with varied structure and content.
- Limitations on system performance and analytics intensity. This is a barrier when a lot of computational intensity is required, such as for large-scale pattern analyses, complex heuristic algorithms and optimization problems. Performance limitations are also a challenge for applications restricted by the speed of data streaming, access latency or data availability.
- Lack of parallelization. Business issues that can be broken down into smaller units of work can be executed simultaneously, improving performance through data or task parallelism.
Complement Rather Than Complete
Big data methods and technologies can bring definite benefits to any government organization. Data warehousing leaders such as Teradata are incorporating these technologies into hybrid big data architectures. (See “Integrate Solutions,” above.) At the same time, it would be naïve to presume that the significant investment in an existing system infrastructure and years of application development could be simply replaced with any new technology, let alone by largely unsecure, open-source products running on commodity components.
David Loshin is the president of Knowledge Integrity. He is also a thought leader, analyst, consultant and author.