Enriching the Basic DataFrame Program
Let’s add a DB and produce more meaningful information using the DataFrame API.
Adding design into the mix
Documentation tends to be a reference, manual, and reminder of an application’s design to consult back and forth. It can also serve as a fundamental blueprint for a project being built, illustrating the landscape and interaction between its parts or components.
As we enrich our application in this lesson by adding a Database into the mix and applying a transformation to the data through the DataFrame API, it is helpful to illustrate how the big picture looks for our application’s changes and, more importantly, how Spark fits into it.
During the duration of this course, we’ll try to enforce this practice, especially when our main batch application increases in its complexity.
The application’s model
Because our application is not complex, the following diagram mixes both architectural and flow perspectives into one:
Get hands-on with 1400+ tech skills courses.