Introduction

Introduction

Spark, the ubiquitous platform for data processing, and has taken over the traditional MapReduce framework. Some technologists go so far as to declare MapReduce dead. Spark outperforms MapReduce by several orders of magnitude in numerous benchmarks and performance studies. Spark was started as a project in 2009 at University of California Berkeley and a research paper on the findings was published the following year. Later, the researchers created the company Databricks which focuses on Spark-based machine learning and analytics solutions. Spark has become the de-facto platform for processing data, for use-cases ranging from batch processing to interactive ad-hoc query analysis. Spark is primarily written in Scala, its default language. But, Spark works with Python, Java, and R.