Rise of Big Data

Rise of Big Data

In today’s world, potential sources of data are all around us and easier than ever to capture digitally. Newspapers, a smart-phone’s GPS location, restaurants we frequently visit, people we meet, our purchases, and when we wake-up and go to sleep, are all pieces of data that can be collected. Several technology advancements makes collecting, storing, and processing ever-increasing amounts of data possible. For example, sensors for a wide variety of tasks, like motion-detection, temperature, wind-speed and others, are cheap and have the capability to periodically emit data back to servers. Smartphones have contributed immensely to the data volume being generated today. Because of faster speed and better network bandwidth, transmitting huge amounts of data is trivial. The drop in cost to procure disk storage and memory and the increased size of these storage mediums have contributed to new opportunities. They have the potential to solve problems that previously seemed too expensive or simply impossible.

The collection of data isn’t an end in itself, but rather it serves as a means to several possible ends. Data is valuable when it can be aggregated, sliced, or diced to extract meaningful patterns, insights, or predictions. For example, Netflix may collect information on movies watched by a user and the movies watched by users with similar demographics and behavior. Then Netflix, can run that information through a recommendation algorithm to make effective recommendations. Google’s omniscient search bar is a wonder of collecting the entirety of the web and then indexing it for speedy retrieval.

The trend of data generated from humans and machines alike, is likely to continue increasing. The mountains of collected data can’t be processed with traditional data stores and processing techniques. It has given way to a new ecosystem of Big Data technologies. Even Artificial Intelligence and Machine Learning benefit from Big Data. AI needs data to build upon its intelligence and more data can better train learning algorithms.

Big Data’s benefits are being harnessed by industries across the board. In the financial world, Big Data detects credit card fraud and collects trade analytics used in high frequency trading. Manufacturers improve operational efficiency by using Big Data to predict machine breakdowns and component replacements. Health care workers do genomic research with Big Data. Oil exploration utilizes Big Data to discover new sites with seismic sensor data. There are countless examples and applications of Big Data in almost every industry. As the Big Data matures, we may see the introduction of new technologies and tools. But we can constantly unlock new use cases. Big Data will continue to allow us to collect and process huge mounds of data.