Spark Environments

There are a variety of ways to both configure Spark clusters and submit commands to a cluster for execution. When getting started with PySpark as a data scientist, my recommendation is to use a freely-available notebook environment for getting up and running with Spark as quickly as possible. While PySpark may not perform quite as well as Java or Scala for large-scale workflows, the ease of development in an interactive programming environment is worth the trade-off.