Big Data Business Analytics, Storage, Integration, Challenges and Opportunities
As the life is getting complex day by day, big data is going to be mismanaged and creating a lot of challenges and obstacles for technical people to analyse, store and integrate this big data which may be in several formats like text, video, image, binary, xml etc. Following article discusses why big data is complex and hard to manage? What are the technical challenges and obstacles we are facing to analyse, integrate and store this big data? What are the future opportunities which big data is creating for us? Lets have this big data discussion in detail:
Big Data is really very big and complex
Lets have a simple example to understand how big data is really very big and complex.
Consider a simple trip to a child’s birthday party. You send a tweet that you’re headed to the party and you create data. You get in the car, stop to get gas, pay at the pump and you create data. You buy a card at the store, scan your frequent shopper card, pay with cash and you create data. You take pictures and a short video at the party, post them on Facebook, Flickr and YouTube and you create data. You send a text message while at the party and you create data. Throughout the entire trip, your cell phone creates data as it continually sends out GPS signals and your car creates data as it tracks fuel efficiency. Take the data for this one activity, multiply it by the number of activities you have, multiply that by the number of people who have activities, and you probably have only a small fraction of the data that’s constantly being generated.
According to IBM, we create 2.5 quintillion bytes of data every day. Ninety percent of the data we have has been created in the past two years and the amount of data is expected to increase exponentially. The data we create is expanding rapidly as enterprises capture more data in greater detail, as multimedia becomes more common, as social media conversations explode and as we use the Internet to get things done. This is “big data,” and it’s getting even bigger.
How is Big Data complex?
Big data is complex. It’s complex because of the variety of data that it encompasses – from structured data, such as transactions we make or measurements we calculate and store, to unstructured data such as text conversations, multimedia presentations and video streams. Big data is complex because of the speed at which it’s delivered and used, such as in “real-time.” And obviously, big data is complex because of the volume of information we are creating. We used to speak in terms of megabytes and gigabytes of home storage – now we speak in terms of terabytes. Enterprises speak in terms of petabytes.
Big Data Market
"Big data" has increased the demand of information management specialists in that Software AG, Oracle Corporation, IBM, Microsoft, SAP, EMC, and HP have spent more than $15 billion on software firms only specializing in data management and analytics. In 2010, this industry on its own was worth more than $100 billion and was growing at almost 10 percent a year: about twice as fast as the software business as a whole.
Developed economies make increasing use of data-intensive technologies. There are 4.6 billion mobile-phone subscriptions worldwide and there are between 1 billion and 2 billion people accessing the internet. Between 1990 and 2005, more than 1 billion people worldwide entered the middle class which means more and more people who gain money will become more literate which in turn leads to information growth. The world's effective capacity to exchange information through telecommunication networks was 281 petabytes in 1986, 471 petabytes in 1993, 2.2 exabytes in 2000, 65 exabytes in 2007 and it is predicted that the amount of traffic flowing over the internet will reach 667 exabytes annually by 2013.
Big Data Analytics
Big data requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times. The practitioners of big data analytics processes are generally hostile to slower shared storage, preferring direct-attached storage (DAS) in its various forms from solid state disk (SSD) to high capacity SATA disk buried inside parallel processing nodes. The perception of shared storage architectures—SAN and NAS—is that they are relatively slow, complex, and expensive. These qualities are not consistent with big data analytics systems that thrive on system performance, commodity infrastructure, and low cost.
Real or near-real time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in memory is good—data on spinning disk at the other end of a FC SAN connection is not. The cost of a SAN at the scale needed for analytics applications is very much higher than other storage techniques.
There are advantages as well as disadvantages to shared storage in big data analytics, but big data analytics practitioners as of 2011 did not favour it.
Big Data Challenges and Obstacles
Big data presents a number of challenges relating to its complexity.
1. One challenge is how we can understand and use big data when it comes in an unstructured format, such as text or video.
2. Another challenge is how we can capture the most important data as it happens and deliver that to the right people in real-time.
3. A third challenge is how we can store the data, and how we can analyze and understand it given its size and our computational capacity.
4. Scarce talent
One topic that came up frequently during the discussions is the shortage of Big Data talent. The good news is that a growing number of Data Science undergraduate, graduate, and certificate programs are launching around the world. A corporate-based model for building Big Data talent can be the more expedient route for countering the issue of talent scarcity. Corporate/higher education collaboration is key to the success of such programs.
5. And there are numerous other challenges, from privacy and security to access and deployment.
According to the latest report Mind the Digital Marketing Gap, produced by the Economist Intelligence Unit (EIU) in partnership with digital marketing software vendor Lyris, 37 per cent of marketing executive surveyed see their difficulty in interpreting big data as the biggest challenge to an effective marketing strategy, while 45 per cent claim to lack sufficient big data analytics capabilities.
In addition, just 24 per cent of marketers said they always use data analysis to develop actionable insights for their overall marketing strategy, although the majority of respondents use data analytics at least some of the time for actionable insights and to personalise consumer communications.
Other key obstacles include a lack of financial resources (43 per cent of respondents); too much emphasis on digital tools and social media; the proliferation of channels; and inadequate human resources (33 per cent apiece).
Big Data Opportunities
But even greater than the challenges are the opportunities that big data presents. Big data is the next frontier for innovation, competition and productivity. We can answer questions with big data that were beyond reach in the past. We can extract insight and knowledge, identify trends and use the data to improve productivity, gain competitive advantage and create substantial value for the world economy. The challenges with big data are limited
compared to the potential benefits, which are limited only by our creativity and ability to make connections among the trillions of bytes of data we have access to.
Big data provides an opportunity to find insight in new and emerging types of data. How will you take advantage of this opportunity?
