Single node spark notebook setup using docker

By Niraj Zade | 2023-11-18   | Tags: guide local pyspark dev env

You need an immediately-available spark notebook setup, to quickly carry out small experiments to clear your doubts. This is the quickest way I know of getting a pyspark notebook up and running within seconds (without headaches).

You'll only need docker, and around 5gb of internet.


If you're on linux:

# create directory to mount into
mkdir -p ~/docker-mounts/pyspark-notebook

# start pyspark notebook container, and sync it to the mount directory 
docker run -it --rm -p 8888:8888 -v ${HOME}/docker-mounts/pyspark-notebook:/home/jovyan --user root -e CHOWN_HOME=yes -e CHOWN_HOME_OPTS='-R' jupyter/pyspark-notebook jupyter notebook --NotebookApp.token=''

It will pull a docker image that eats up 4.91GB of disk space

If you're on windows, I don't know how to get this command working on it (I don't use windows).

Specifically, I don't know how to set -

  • -v - the docker mounts
  • --user - used to fix written file permissions

If you get the command working, email me and I'll put it up here.

Accessing the notebooks

You can open the notebook server on:


  • Everything in the home directory is saved to disk.
  • I've disabled the authentication with the --NotebookApp.token='' flag.
  • It runs spark in standalone mode. Good enough for doing small experiments.

That's it. Enjoy.

© 2024 | Website was autogenerated on 2024-04-24

Brought to you by Niraj Zade - Website, Linkedin

~ whoever owns storage, owns computing ~