pySparkGuide.com

pySparkGuide.com

Reference and tutorials for busy Big Data Professionals.
The finest pySpark resource on the internet!

Misc

guide - Single node spark notebook setup using docker

performance

spark - Spark Adaptive Query Engine (AQE) - all the details you need to know
spark - Spark join strategies

pySpark api

guide - Cast string to Date or DateTime using to_date() and to_timestamp()

pyspark

theory - SQL - Note on window restrictions for window functions
theory - api - SQL Expressions vs Dataframe API expressions in pySpark functions
guide - perf - Cache and persist - why and how
guide - Style guide - Chaining functions in python
theory - io - Understanding partition discovery and partition read optimization in spark

read and write

guide - JSON - read, set schema and write with pySpark
guide - Delta - read, set schema and write with pySpark
theory - Read and write modes in pySpark
guide - CSV - read, set schema and write with pySpark

sql