Running Python Jupyter Notebooks Using Live VM
This is a step-by-step guide to build and run Python Jupyter Notebooks inside a Live VM.
If you have existing code, text, or other material in the form of Jupyter notebooks, you may be wondering if there’s a way to embed your .ipynb
files directly into an Educative course. Our platform allows you to embed your Jupyter notebooks in your lessons within a live virtual machine (Live VM) by using our Live App widget.
Overview
To enable this advanced functionality, you will first need to perform a one-time Docker setup. The result will be a Docker environment that supports Jupyter notebooks in your course. You will then be able to select and embed your .ipynb
files in individual lessons.
Here is an overview of the setup steps you will perform:
-
Create a Dockerfile.
-
Prepare a Dockerfile tarball (including all your
.ipynb
files) for upload. -
Upload the Dockerfile tarball to your course in the Docker Container section.
-
Configure a Jupyter Docker job to activate an
.ipynb
file of your choice in the Live App widget. -
Embed a Live App in a lesson of your choice, linked to the Docker job.
Note: If you are unfamiliar with Docker, don’t worry. You can simply follow the instructions below. If you’d first like additional context, read our Custom Docker Environments section and then return to this lesson.
Let’s go over each of these steps in detail.
Prepare Your Dockerfile and Python Jupyter Files
To embed Python Jupyter notebooks in your lessons, you will first need to set up a custom Docker environment for use in your course. In order to do that, you will need to prepare a zipped tarball that includes three things:
- A Dockerfile with instructions needed to install Python and Jupyter Labs in Docker
- Your
.ipynb
files - A Python configuration file,
config.py
These resources are combined into a zipped tarball, which is then used to build a Docker image later on.
First, let’s create our Dockerfile.
Create a Dockerfile for Python Jupyter
Note: You have to perform this step locally (on your computer).
- Create a Dockerfile by adding the following script to an empty text file saved as
Dockerfile
(with no extension):
# 20.04 version of ubuntu as base imageFROM ubuntu:20.04# Install the following packagesRUN apt-get update && apt-get install software-properties-common -y &&\add-apt-repository ppa:deadsnakes/ppa && apt-get update &&\apt-get install python3.6 -y && apt install python3-pip -y &&\pip3 install --upgrade pip && pip3 install jupyter &&\mkdir /usr/local/notebooks# Install any Python modules required in the notebookRUN pip3 install pandas && pip3 install numpy && pip3 install seaborn &&\pip3 install matplotlib && pip3 install sklearn# Add configuration fileADD config.py /root/.jupyter/jupyter_notebook_config.py# Add ipynb filesADD helloworld.ipynb /usr/local/notebooks/
You can copy the above script into your Dockerfile.
Dockerfile Explained
Docker uses the instructions in your Dockerfile to build a Docker image. This image will be used to run your Jupyter notebooks. Let’s take a look at what our Dockerfile contains.
-
FROM ubuntu:20.04
- We are using the 20.04 version of ubuntu as base image of our Dockerfile. -
RUN ... install software-properties-common -y
Ubuntu uses the package manager
apt
. This allows you to easily install any packages or software required in your environment.-
add-apt-repository ppa:deadsnakes/ppa
- add-apt-repository - This will add the specified apt repository to the list.
- ppa:deadsnakes/ppa - This will add the PPA deadsnakes repository, which contains the
python3.6
package.
-
apt-get update
- This will check the dependencies of the packages you want and install any that are needed. -
apt-get install python3.6
- This will install Python 3.6. -
install python3-pip -y
- This will install the Python package managerpip
which is needed in order to install JupyterLab. -
pip3 install --upgrade pip
- This will upgradepip
. -
pip3 install jupyterlab
- This will install theJupyterLab
package for Python. -
mkdir /usr/local/notebooks
- This is the directory where you will add youripynb
files.
-
-
RUN pip3 ... install sklearn
This will install any Python modules that you need in your Jupyter notebooks.
-
ADD config.py /root/.jupyter/jupyter_notebook_config.py
This will add the configuration file for your notebooks. It contains preferences such as the port number (set to
8080
) and the notebook directory (/usr/local/notebooks
). -
ADD helloworld.ipynb /usr/local/notebooks/
This will add your
ipynb
files to your notebook directory. We have addedhelloworld.ipynb
as an example.
We are now ready to compress our Dockerfile into a tarball, which will be used later on below.
Compress Your Files into a Tarball File
Next, you need to compress the Dockerfile, your .ipynb
files, and the config.py
file into a single tarball file. Here’s how:
- Use your command line terminal to navigate to the directory where your
Dockerfile
is- Move all the
.ipynb
files your course needs into this directory as well. - Also move the
config.py
file into this directory.
- Move all the
- In your terminal, use the following command to turn these assets into a tarball:
tar -czvf jupyter.tar.gz Dockerfile helloworld.ipynb config.py
This command will take your Dockerfile
, .ipynb
files, and config.py
file as inputs and create a single tarball file as output. The tarball file will be named jupyter.tar.gz
.
Note: In our example, we are only using one
.ipynb
file:helloworld.ipynb
.
For your convenience, the files are provided below. Please download them.
c.NotebookApp.port = 8080 c.NotebookApp.ip = '0.0.0.0' # listen on all IPs c.NotebookApp.token = '' c.NotebookApp.allow_origin = '*' #allow all origins c.NotebookApp.notebook_dir = '/usr/local/notebooks' c.NotebookApp.disable_check_xsrf = True c.NotebookApp.tornado_settings = { 'headers': { 'Content-Security-Policy': "frame-ancestors https://*.appspot.com https://*.educative.io 'self' ;", } }
Now that you have the files, create the tarball, upload it and configure your Docker job!
Upload the Tarball
Go to your course editor page. Navigate to the Docker Container section. It looks like this:
Click on Change Docker File (Beta). Then select the tarball jupyter.tar.gz
.
Click on the Save button to start the docker image building process.
Wait for the Image to Build
Building an image will take a few minutes. If your image is built successfully, you will see the following message:
However, if your message build has failed, you will see the Image build failed message displayed instead.
If the build fails, check the build logs:
The logs will indicate where the building process failed. It is a great way to debug your Dockerfile.
If all else fails, send us a message at authors@educative.io.
Create the Docker Job
The next step is to configure a Jupyter Docker job—think of this as some extra Jupyter-specific settings that are needed for the LiveApp widget to work properly later.
After the image successfully builds, click the + sign in the Docker Container section. The screen will expand to show this:
Use the following settings:
Job Name
You can include as many Docker jobs in a course as you need. The Job Name acts as an ID to each job. In this example, we are setting up a job named jupyter
.
Input File Name
In our Live VM functionality, this field can have any value in it. For demonstration purposes, let’s call it helloworld
.
Run Script
The Run Script is the most important part of the job. This script is run every time the Live App is launched. In our case, we want to launch the helloworld.ipynb
in Jupyter. Add the lines given below in your Run Script:
nohup jupyter notebook /usr/local/notebooks/helloworld.ipynb --allow-root --no-browser > /dev/null 2>&1 &
Run in Live Container
Check this box to enable LiveVM functionality.
Application Port
This port must match the port specified in config.py
. In our case, it is 8080
.
Start Script
The Start Script is not required for the Live App widget, but it cannot be left empty. Hence, a simple echo
command should be fine:
echo "hello world"
Putting it All Together
Your final Jupyter job will look like this:
Our Docker setup is now ready! Let’s use it in a Live App
Use Jupyter Notebook in Live App
Everything we’ve done up to this point has been to embed a Python Jupyter Notebook in our lessons. As our last step, we need to insert a Live App widget in our lesson and choose the Docker job.
A blank Live App widget will now appear. You will also see two fields: App Entrypoint and Live Docker Job:
App Entrypoint
The App Entrypoint displays the URL of your container. From this URL, we must navigate to the ipynb
file in our notebooks
directory.
In our case, the required file is /notebooks/helloworld.ipynb
. Add this at the end of the URL.
Now the URL will https://jy9lnm-live-app.educative.run/notebooks/helloworld.ipynb
.
Note: For each
ipynb
file in your tarball, you will have to create a separate Docker job, and specify the file’s path in the App Entrypoint field, e.g,URL/notebooks.filename.ipynb
.
Live Docker Job
In the Live Docker Job field, select the Docker job we created: jupyter
.
Click the “Click to launch app!” 🚀 button and you will now see helloworld.ipynb
running as a Python Jupyter notebook.
A Working Jupyter Notebook
A demo of our working Jupyter notebook can be found here.
Kernel Error
In your Jupyter Notebook setup, you may encounter a kernel error:
To fix this, you can follow the steps below.
- Update the
config.py
. Assign empty values totoken
andpassword
. Setdisable_check_xsrf
equal totrue
:
c.NotebookApp.port = 8080c.NotebookApp.ip = '0.0.0.0' # listen on all IPsc.NotebookApp.token = ''c.NotebookApp.password = ''c.NotebookApp.allow_origin = '*' #allow all originsc.NotebookApp.notebook_dir = '/usr/local/notebooks'c.NotebookApp.disable_check_xsrf = Truec.NotebookApp.tornado_settings = {'headers': {'Content-Security-Policy': "frame-ancestors https://*.appspot.com https://*.educative.io 'self' ;",}}
- Lock the JupyterLab version in the Dockerfile:
RUN pip3 install jupyterlab==3.0.1
Conclusion
Congratulations! You have now embedded a Python Jupyter Notebooks into your course by using our Live App widget and selecting a specific .ipynb
file, which you uploaded during the Docker setup process.
Although we used the helloworld.ipynb
notebook as an example, you can embed your own .ipynb
notebook files in your lessons. Simply include the Jupyter notebook files you need in your tarball.
Then you can embed your notebooks, one per Live App widget, by running it in the Docker job’s Run Script and adding its path to the App Entrypoint URL (/notebooks/filename.ipynb
).
Miscellaneous Notes
📝 Note: The Live App session is active only for 15 minutes. During that time, all files that were added in the tarball can be accessed. However, a new file can be created at run time, but this will reset once the session becomes inactive. The following activities on the file are session-dependent:
- The existence of a new file created at run time.
- Any change (update or delete) to the existing files in the tar, or to a new file that was created at run time.
If you have any trouble setting up your Python Jupyter notebooks, send us a message at authors@educative.io.