Running a PySpark Project using Live App

We'll cover the following

PySpark

Docker job for Live Application Widget (Jupyter Notebook)

Select Docker job type
Job name
Input file name
Run script
Application port
Start script

Select Docker job and Override entry point
Let’s run PySpark code in Live Application widget

Let’s see what each field in the above job means:

Select Docker job type

This is Docker Job Type selection in which we have to select what kind of docker job we are creating.

Live

Job name

This is just a job name for reference. You can use any name you want to specify for this job.

PySparkLive

Input file name

Name of the input file you want to run in the live widget. We are running the Jupyter Notebook server, so in our case, we will add a pseudo name here.

foo

Run script

This script runs when we execute the code in the live widget. It is mandatory.

nohup jupyter notebook /usr/local/notebooks/helloworld.ipynb --allow-root --no-browser > /dev/null 2>&1 &

Application port

We have to specify the port on which we want to run it.

Start script

This script runs when we execute the Live Widget for the very first time. It is mandatory.

echo "Hello World"

Select Docker job and Override entry point

After creating the docker job for the Live Application widget now select it and override the entry point as given below.

__default