|By Jill Tummler Singer||
|October 15, 2012 08:15 AM EDT||
Everyone's talking about Big Data today. Is Big Data a buzz word, real phenomenon, or next evolution in our world? In this AFCEA white paper, we surveyed many notable experts to gain perspectives on Big Data. This paper serves as a primer on Big Data characteristics and provides insights into technology challenges and solutions. The intent is to help federal agencies, companies, and communities develop new solutions for consuming, storing, processing, and analyzing Big Data in order to find the right answers needed to accomplish the mission, gain competitive advantage, and collaborate in more meaningful ways.
It is a BIG DATA world with a current volume of 1.8 zetabytes of data created per year and doubling every two years. Technologies--cloud computing, Hadoop, MapReduce, flash array storage, business intelligence tools, etc-- are vital to an organization's ability to keep pace with Big Data. The Big Data architecture introduces a fourth layer in the cloud computing stack. Knowledge as a Service joins the traditional cloud layers (Infrastructure as a Service, Platform as a Service, and Software as a Service) as a focused layer dedicated to the management and analytics of Big Data including binding concepts such as pedigree, lineage, and provenance of data. Harnessing Big Data will require a combination of technology implementations, business process changes, and workforce training to achieve breakthroughs for your organization.
What Is Big Data?
Let's start with a simple definition. McKinsey and Company define Big Data as "Datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze". Big Data can also be characterized by a series of descriptors starting with the letter V. In the 1990s, the word "volume" emerged to describe the rapidly growing rate of data in the Internet age. The table below, from Barbara Wixom of the University of Virginia, offers 15 different data dimensions for Big Data. For brevity's sake, let's examine the four most common descriptors of volume, velocity, variety, and validity.
Source: Barbara Wixom, 2012
The volume of data produced in a 24-hour period is staggering and amounts to 1.8 zetabytes per year. Every day, 2 million blogs are posted, 172 million users visit Facebook (spending a combined 4.7 billion minutes on a single social networking site), 51 million minutes of video are uploaded, and 250 million digital photos are shared. We continue to generate 294 billion emails each day, even though many consider email an outdated form of communication.
Perhaps more fascinating is that data velocity is accelerating. Velocity is the speed at which data is growing and this extreme speed (1.8 zettabytes now and 3.6 zettabytes in 2013) is taxing our current information technology capabilities. According to an IDC Digital Universe Study, we are doubling the world's information every 18 months. This trend will not slow down anytime soon. Will we be able to manage data in our near future? Did you know that each second of high-definition video generates two thousand times as many bytes as one single page of text? IBM research indicates that 90 percent of the world's data has been created in the last two years alone. Apple is selling more iPhones per day than they are babies born in the world, as noted by MBAOnline.com. Samsung's smart phone sales are ahead of Apple sales at an estimated 41 million to 32.6 million for 2Q2012, and demonstrate the rise of additional handheld platforms contributing to the speed at which we create new data.
The third descriptor for Big Data is variety or the types of data being created. You can generally split variety into structured and unstructured form. The 294 billion emails per day can be considered structured text and one of the simplest forms of Big Data. Financial transactions including movie ticket sales, gasoline sales, restaurant sales, etc., are generally structured and make up a small fraction of the data running around the global networks today. Unstructured data is a primary source of growth in variety. Music is an ever increasing variety of data and we are streaming nearly 19 million hours of music each day over the free music service, Pandora. Spotify, a paid streaming media service, is now the number two revenue source for music labels-second behind Apple's iTunes. Old television shows and movies are another source of variety in the non-structured realm. There are over 864,000 hours of video uploaded to YouTube each day. According to MBAOnline.com, we could pipe 98 years of non-stop cat videos into everyone's home for endless hours of boredom, fun, or insanity!
The biggest challenge from a data variety perspective is harnessing the unstructured information for business relevance and data driven decisions. We've spent decades perfecting analytic tools for structured information. Analytic tools for unstructured data are more limited and less intuitive. Not all relevant marketing data, as an example, is confined to structured business transactions. Tweets, Facebook posts, YouTube video, and so forth, now represent valid indicators to a business. Corporate reputations can be improved or demolished nearly instantaneously by these new sources of data.
Validity is a singular term designed to characterize the quality, pedigree, lineage, provenance, value, integrity, setting, and context for the data. Structured and unstructured data needs validity characterization and it should follow the data from acquisition to retirement. Information from a trusted source is more highly valued than information from a new or casual source. Over time, however, the new source can be further tested and validity of prior data from that source may increase or decrease. Maintaining data heritage is more complex than meta-data tagging and requires situational awareness (i.e., context) when using the data for business decisions.
Is Cloud Computing ready for Big Data? Is IT ready for Big Data?
Our survey respondents (notable experts in the Big Data, Cloud Computing, and IT industry) generally concurred on the need for fast, flexible IT infrastructure to support Big Data. Anything that takes the infrastructure challenges out of the way of the business was deemed part of the critical path to success. Tim Estes, Digital Reasoning, noted the cloud "gives you the speed you need". Similarly, Jeff Jonas from IBM declared any infrastructure that lets you "scale up and out affordably is goodness." Four specific areas were identified for deeper investigation as we considered if Cloud Computing and IT departments were ready for Big Data: security; store and process; sensemaking; and stewardship.
Security remains the number one obstacle to preventing IT organizations from adopting Cloud Computing. This same AFCEA sub-committee explored security in cloud computing in a white paper released in 2011. The full white paper can be reviewed through this link.
In summary, the 2011 AFCEA white paper noted that cloud computing offers significantly improved visibility and insight that drives new cyber security solutions. Access to cloud computing services in traditional computing environments and in modern mobile environments provides numerous opportunities to gain visibility and retrieve security data points across your infrastructure, platforms, and applications. Collecting pulse points from the high-speed networks used to connect to your cloud provides insight into threats attempting to breach the perimeter of your infrastructure. Remote access devices and global position/location can be detected through other data points, triggering the requirement for additional security access and authorization controls while also providing real-time knowledge of the security status of end-user devices. Constant monitoring of applications and platforms offers additional data collection points for discovering vulnerabilities in applications that can be used to infiltrate the infrastructure. Moreover, merging measures and metrics from co-located environments or other cloud locations in your global enterprise can add yet another layer of data to the collection.
Store and Process
A May 2012 Meritalk Study called "The Big Data Gap" noted that most government IT leaders are fairly positive about the storage and processing resources needed to harness Big Data. The study indicates the respondents currently own about half of the compute resources they actually need. The fear is they have only about 20 percent of the capacity (in storage and processing power) needed to manage the Big Data headed their way. Industry cloud providers have developed efficient Hadoop cloud-based architectures to handle Big Data. Left on its own, Hadoop will very efficiently process your Big Data but will do so leaving no resources available for other work. Experience indicates restrictions are required on Hadoop jobs to prevent Hadoop from grabbing all available storage and processing resources from the cloud. Industry advances for storage and processing are rapidly emerging to respond to Big Data requirements. Flash array storage is now available at enterprise-class levels from companies like Whiptail and EMC. The InfiniBand trade association created a high performance computing input/output fabric to deliver the internal data center speed needed for Big Data. Gaming providers like NVidia have implemented high-speed clouds for acceleration using graphical processing units (GPU). Moreover, global network providers continue to drive performance advances in optical and electrical equipment with a goal of keeping pace with the explosion of streaming bits and bytes. Cisco's Global Forecast predicts global network traffic will exceed 110 Exabytes per month by 2016. More answers and technologies are emerging each day to overcome remaining deficiencies in storage and processing capabilities for Big Data.
In the world of Big Data, making sense of the data is not trivial. The volume, velocity, variety, and validity of the data now available can create link analysis diagrams more closely resembling nature's most intricate floral design. You would need Sheldon Cooper's Big Bang Theory eidetic memory to begin to make sense of the data...not to mention a movie theater-sized computer monitor screen! What if your primary viewing device is a smart phone or a 5 inch tablet? Both Gartner and Forrester predict significant increases in the use of mobile devices as Business Intelligence access platforms.
What you need are solutions that make sense of new data in time to derive new observations as the observations actually happen. You also need solutions that allow you to make decisions fast enough to do something about the old and new observations while the transaction is still happening. The tools must protect various slices of the complex data to ensure the user is approved to see the underlying data sources based on the provenance of each individual data element. Advanced analytic and security tools to separate data appropriately, whether your data is in a public cloud or a private cloud, are necessary. Commercial cloud providers are keenly focused on data and privacy protection to ensure only authorized users gain access--using technologies such as encryption, identity management, authorization services, etc. Commercial cloud providers are accumulating very successful track records for safeguarding information.
The fourth challenge for the IT department is data stewardship-often considered data ownership and/or data management. The Big Data Gap MeriTalk survey from May 2012 indicated that nearly 70 percent of respondents thought the "IT Department" had some role in owning and managing the data. Less than 30 percent of respondents indicated the department generating the data should have these roles. Regardless of the decision, someone has to own and manage the data.
The link between financial performance and effective data management is strengthening as companies learn to harvest Big Data. The Economist Intelligence Unit indicates strategies for collecting and analyzing data need to rise to the C-suite level. In essence, Big Data makes IT even more strategic to the business. New/modified career tracks are emerging (in forward-leaning organizations) to develop employees with the skills necessary to validate data sources, define and measure pedigree and lineage of data sources, and effectively manage Big Data for an enterprise.
Winning in a Big Data World
As noted above, Big Data is complex and the challenges it presents are daunting. In today's environment, Big Data demands phenomenal corporate balance. Success and competitive advantage require you to focus on technologies, business processes, and people. Your company needs:
- New technologies for controlling Big Data;
- Business processes designed for rapid decisions using Big Data; and
- People trained to make smart decisions exploiting Big Data.
The Cloud Computing technology stack is evolving to handle Big Data. Our survey revealed the need for a new, four layer stack associated with a Big Data architecture. The Cloud stack of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS) must shift to accommodate the introduction of Knowledge as a Service (KaaS) in between PaaS and SaaS. This figure depicts the insertion of the new KaaS layer to drive Big Data results. Without a Knowledge layer, companies end up investing heavily in customer knowledge engineering, adaptors, and connectors--essentially ending the elastic advantage of the Cloud at the IaaS layer. This slows application development and requires human-intensive work on data maps/models and ontologies across disparate systems. The KaaS layer pushes you into a heavily automated, algorithm-driven common knowledge layer that embraces Cloud speed and elasticity.
Innovators have embraced this KaaS idea with Google's PageRank algorithm as an example of a common index with searchable attributes capable of working across the diversity of the Web without relying on substantial amounts of manual organization. Deep insights from Big Data require semantic silos-both structured and unstructured content--to be "knowledge processed" and moved into the Cloud in order to capitalize on the business advantages opportunities of data mining. The specific technologies to ingest and hold structured and unstructured data (as repositories) are similar with any processing differences derived from the complexity of business needs. The repositories are living; as new data arrives, its connection to existing data will be identified and may new data may change the value of the old data to your business. This concept of "data finding data" is the subject of IBM efforts under the direction of Jeff Jonas. You can read more at the here.
In a Big Data world, companies should pay as much attention to the data as they do their other corporate assets, e.g., labor and capital. Company goals must be clear with labor, capital, and data aligned. Your employees should be empowered to make decisions with proper checks, balances, and audits built in. Speed is critical and using multi-layered and lengthy paths to finalize decisions will limit your competitive advantage. Business processes designed around lessons learned and adaptability will facilitate Big Data organizations. Governance models and decision thresholds for employees should be clear, with escalation paths obvious and some understanding of the potential "mosaic effect" present. The "mosaic effect" occurs when seemingly unclassified or benign data are combined together by an analyst with a resulting picture that becomes confidential or more highly classified. Training your employees to recognize the "mosaic effect" is necessary to protect sensitive results, intellectual property, and competitive advantage. Finally, adhering to strict data management rules (e.g., process once and use many times), will facilitate long-term Big Data utility by protecting the integrity and provenance of your corporate data.
The third element in your winning balance in a Big Data world is your people. Bad decisions in today's world are more obvious and less tolerated. Sometimes, analysis of Big Data drives a company to the wrong conclusion or decision. Other times, Big Data can be successfully used to back up instincts with facts. Big Data in financial institutions has become so complex, the Federal Deposit Insurance Corporation created an entire office for overseeing Complex Financial Institutions (FDIC/CFI).
Companies that successfully train their people to use Big Data wisely will reap financial and market share rewards. Alternative analysis, critical thinking, and other analytic skills-combined with experiential learning and mentoring--will be necessary to ensure your team is seeing the right answer in the possibilities. Thornton May's book, The New Know, highlights the requirement for companies to capitalize on data and brain power to have good knowledge of what happens next...not what happened in the past. Generally speaking, everybody will have a Moneyball team in the future. What's going to set your team apart?
Big Data Summary
Cloud Computing provides the technology foundation to capitalize on Big Data for corporate success. The flexible infrastructure offered through Cloud Computing--combined with increased storage and processing power of new technologies in the Cloud-provide the rich, agile compute platform to handle the volume, variety, velocity, and validity needs of Big Data. Insert a fourth layer into the Cloud Computing stack (Knowledge as a Service) between PaaS and Saas to reduce human-intensive ontology work in favor of automated, algorithm-driven features designed to exploit disparate data in the Cloud.
Beyond technology, Big Data is likely to require changes in your business processes to ensure decisions with proper analytic judgment with necessary oversight operating at the right speed for competitive advantage. Spend time training your people to analyze data from alternative points of view and to quickly accept automatically-generated observations. People make decisions; data doesn't. Drowning a poorly trained employee in loads of data will still produce poor decisions.
Social media, instantaneous access, and an "always connected" population of stakeholders will increasingly demand transparent accountability. Use Big Data through Cloud Computing to demonstrate your corporate decisions are clearly backed by facts.
Cloud and Big Data Experts Surveyed
- Zalmai Azmi, Senior Vice President, CACI International, Inc.
- Charles Croom, Vice President, Lockheed Martin Information Technology
- Christopher Day, Senior Vice President, Terremark Federal Group (A Verizon Company)
- John Dvorak, Federal Bureau of Investigation
- Tim Estes, Chairman and CEO, Digital Reasoning Systems
- Jeff Jonas, Chief Scientist, IBM
- Barbara Wixon, Associate Professor of Commerce, University of Virginia
Other Sources Cited and/or Used
- How Much Data Will Humans Create & Store This Year, Josh Catone (June 28, 2011)
- Three Big WHATs to Identify Big Data Challenges, Pearl Zhu (April 2012)
- Bringing big data to the enterprise, IBM
- The BIG Picture on BIG DATA, GovConExec (April 2012)
- Big Data: The next frontier for innovation, competition, and productivity, McKinsey Global Institute (May 2011)
- Big Data: Harnessing a game-changing asset, The Economist Intelligence Unit (September 2011)
- A Day in the Life of the Internet, Matt Silverman (March 06, 2012)
- How to Be Ready for Big Data, Thor Olavsrud (March 20, 2012)
- The Big Data Gap, Meritalk (May 7, 2012)
- Cloudy with a Chance of Savings, Meritalk (April 25, 2012)
- The New Know, Thornton May
- Moneyball, Michael Lewis
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Jan. 30, 2015 03:30 AM EST Reads: 5,141
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
Jan. 30, 2015 03:30 AM EST Reads: 5,784
The true value of the Internet of Things (IoT) lies not just in the data, but through the services that protect the data, perform the analysis and present findings in a usable way. With many IoT elements rooted in traditional IT components, Big Data and IoT isn’t just a play for enterprise. In fact, the IoT presents SMBs with the prospect of launching entirely new activities and exploring innovative areas. CompTIA research identifies several areas where IoT is expected to have the greatest impact.
Jan. 30, 2015 03:00 AM EST Reads: 2,797
There is no doubt that Big Data is here and getting bigger every day. Building a Big Data infrastructure today is no easy task. There are an enormous number of choices for database engines and technologies. To make things even more challenging, requirements are getting more sophisticated, and the standard paradigm of supporting historical analytics queries is often just one facet of what is needed. As Big Data growth continues, organizations are demanding real-time access to data, allowing immediate and actionable interpretation of events as they happen. Another aspect concerns how to deliver ...
Jan. 30, 2015 03:00 AM EST Reads: 4,857
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
Jan. 30, 2015 02:30 AM EST Reads: 4,790
Scott Jenson leads a project called The Physical Web within the Chrome team at Google. Project members are working to take the scalability and openness of the web and use it to talk to the exponentially exploding range of smart devices. Nearly every company today working on the IoT comes up with the same basic solution: use my server and you'll be fine. But if we really believe there will be trillions of these devices, that just can't scale. We need a system that is open a scalable and by using the URL as a basic building block, we open this up and get the same resilience that the web enjoys.
Jan. 30, 2015 02:00 AM EST Reads: 4,844
Code Halos - aka "digital fingerprints" - are the key organizing principle to understand a) how dumb things become smart and b) how to monetize this dynamic. In his session at @ThingsExpo, Robert Brown, AVP, Center for the Future of Work at Cognizant Technology Solutions, outlined research, analysis and recommendations from his recently published book on this phenomena on the way leading edge organizations like GE and Disney are unlocking the Internet of Things opportunity and what steps your organization should be taking to position itself for the next platform of digital competition.
Jan. 30, 2015 02:00 AM EST Reads: 4,809
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect at GE, and Ibrahim Gokcen, who leads GE's advanced IoT analytics, focused on the Internet of Things / Industrial Internet and how to make it operational for business end-users. Learn about the challenges posed by machine and sensor data and how to marry it with enterprise data. They also discussed the tips and tricks to provide the Industrial Internet as an end-user consumable service using Big Data Analytics and Industrial Cloud.
Jan. 30, 2015 01:00 AM EST Reads: 4,524
How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happens, where data lives and where the interface lies. For instance, it's a mix of architectural styles ...
Jan. 30, 2015 12:30 AM EST Reads: 4,862
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what the future may hold. Mike Kavis is Vice President & Principal Cloud Architect at Cloud Technology Pa...
Jan. 29, 2015 06:15 PM EST Reads: 5,465
Dale Kim is the Director of Industry Solutions at MapR. His background includes a variety of technical and management roles at information technology companies. While his experience includes work with relational databases, much of his career pertains to non-relational data in the areas of search, content management, and NoSQL, and includes senior roles in technical marketing, sales engineering, and support engineering. Dale holds an MBA from Santa Clara University, and a BA in Computer Science from the University of California, Berkeley.
Jan. 29, 2015 06:00 PM EST Reads: 5,197
The Internet of Things (IoT) is rapidly in the process of breaking from its heretofore relatively obscure enterprise applications (such as plant floor control and supply chain management) and going mainstream into the consumer space. More and more creative folks are interconnecting everyday products such as household items, mobile devices, appliances and cars, and unleashing new and imaginative scenarios. We are seeing a lot of excitement around applications in home automation, personal fitness, and in-car entertainment and this excitement will bleed into other areas. On the commercial side, m...
Jan. 29, 2015 06:00 PM EST Reads: 4,951
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
Jan. 29, 2015 05:00 PM EST Reads: 6,071
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Jan. 29, 2015 02:30 PM EST Reads: 4,415
Performance is the intersection of power, agility, control, and choice. If you value performance, and more specifically consistent performance, you need to look beyond simple virtualized compute. Many factors need to be considered to create a truly performant environment. In his General Session at 15th Cloud Expo, Harold Hannon, Sr. Software Architect at SoftLayer, discussed how to take advantage of a multitude of compute options and platform features to make cloud the cornerstone of your online presence.
Jan. 29, 2015 02:15 PM EST Reads: 5,199
Since 2008 and for the first time in history, more than half of humans live in urban areas, urging cities to become “smart.” Today, cities can leverage the wide availability of smartphones combined with new technologies such as Beacons or NFC to connect their urban furniture and environment to create citizen-first services that improve transportation, way-finding and information delivery. In her session at @ThingsExpo, Laetitia Gazel-Anthoine, CEO of Connecthings, will focus on successful use cases.
Jan. 29, 2015 01:30 PM EST Reads: 3,239
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water, are pursuing SmartGrid initiatives that represent one of the more mature examples of SAE. We have s...
Jan. 29, 2015 01:30 PM EST Reads: 4,882
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
Jan. 29, 2015 01:15 PM EST Reads: 4,261
“With easy-to-use SDKs for Atmel’s platforms, IoT developers can now reap the benefits of realtime communication, and bypass the security pitfalls and configuration complexities that put IoT deployments at risk,” said Todd Greene, founder & CEO of PubNub. PubNub will team with Atmel at CES 2015 to launch full SDK support for Atmel’s MCU, MPU, and Wireless SoC platforms. Atmel developers now have access to PubNub’s secure Publish/Subscribe messaging with guaranteed ¼ second latencies across PubNub’s 14 global points-of-presence. PubNub delivers secure communication through firewalls, proxy ser...
Jan. 29, 2015 12:45 PM EST Reads: 2,478
The industrial software market has treated data with the mentality of “collect everything now, worry about how to use it later.” We now find ourselves buried in data, with the pervasive connectivity of the (Industrial) Internet of Things only piling on more numbers. There’s too much data and not enough information. In his session at @ThingsExpo, Bob Gates, Global Marketing Director, GE’s Intelligent Platforms business, to discuss how realizing the power of IoT, software developers are now focused on understanding how industrial data can create intelligence for industrial operations. Imagine ...
Jan. 29, 2015 12:30 PM EST Reads: 2,867