Tag: theory (6)

Note on window restrictions for window functions
We have a lot of ways to specify a window. However, the PySpark SQL engine only accepts specific kinds of windows for specific window functions. We cannot use any specification of window for any function.
2024-04-16   2 min   theory 6

Spark Adaptive Query Engine (AQE) - all the details you need to know
Spark Adaptive Query Engine - what problems it solves and how it works in detail.
2024-02-20   15 min   theory 6 performance 3

Spark join strategies
Spark join strategies
2024-01-22   16 min   theory 6 performance 3

SQL Expressions vs Dataframe API expressions in pySpark functions
SQL expression and Column expressions used in Spark API
2024-01-16   3 min   theory 6 sql 5 api 2

Read and write modes in pySpark
read and write modes
2023-12-09   2 min   theory 6 reference 2 file-io 4

Understanding partition discovery and partition read optimization in spark
Understanding spark's partition discovery
2023-11-16   6 min   theory 6 partitioning 1