Perks of Apache Spark

Keeps application running

Spark provides graceful degradation in cases where memory is not enough so that the application does not fail but keeps running with decreased performance.

For instance, Spark can recalculate any partitions on demand when they don’t fit in memory or spill them to disk.

Increasing performance

Wide dependencies cause more data to be exchanged between nodes compared to narrow dependencies, so performance is increased significantly by reducing wide dependencies, or the amount of data that needs to be shuffled. One way to do this is by pre-aggregating data, also known as map-side reduction.

Note: As explained previously, the map-side reduction is a capability provided in the MapReduce framework through combiners.

Example

The following code performs a word count in two different ways:

• The first one will send multiple records of value 1 for each word across the network
• The second one will send one record for each word containing the number of occurrences.