pySparkGuide.com

functions

• guide (12) - Cast string column to Date or DateTime using to_date() or to_timestamp()

misc

• theory (6) - SQL Expressions vs Dataframe API expressions in pySpark functions
• guide (12) - Single node spark notebook setup using docker
• guide (12) - Style guide - Chaining functions in python

performance

• guide (12) - Cache and persist - why and how
• theory (6) - Spark Adaptive Query Engine (AQE) - all the details you need to know
• theory (6) - Spark join strategies

read and write

• guide (12) - CSV - read, set schema and write with pySpark
• guide (12) - Delta - read, set schema and write with pySpark
• guide (12) - JSON - read, set schema and write with pySpark
• theory (6) - Read and write modes in pySpark
• theory (6) - Understanding partition discovery and partition read optimization in spark

sql

• guide (12) - CASE in pySpark
• guide (12) - Filter rows in pySpark
• guide (12) - GROUP BY in pySpark
• theory (6) - Note on window restrictions for window functions
• guide (12) - Window functions with pySpark

workflow

• guide (12) - Remove duplicate rows from dataframe