pySparkGuide.com

functions

• guide 12 - Cast string column to Date or DateTime using to_date() or to_timestamp()

misc

• guide 12 - Run spark locally with jupyter notebook
• theory 6 - SQL Expressions vs Dataframe API expressions in pySpark functions
• guide 12 - Style guide - Chaining functions in python

performance

• guide 12 - Cache and persist - why and how
• theory 6 - Spark Adaptive Query Engine (AQE) - all the details you need to know
• theory 6 - Spark join strategies

read and write

• guide 12 - CSV - read, set schema and write with pySpark
• guide 12 - Delta - read, set schema and write with pySpark
• guide 12 - JSON - read, set schema and write with pySpark
• theory 6 - Read and write modes in pySpark
• theory 6 - Understanding partition discovery and partition read optimization in spark

sql

• guide 12 - CASE in pySpark
• guide 12 - Filter rows in pySpark
• guide 12 - GROUP BY in pySpark
• theory 6 - Note on window restrictions for window functions
• guide 12 - Window functions with pySpark

workflow

• guide 12 - Remove duplicate rows from dataframe
• workflow 2 - UDF using python function in PySpark