Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Fundamentals

• An overview of the Big Data ecosystem and Spark's role in contemporary data platforms
• Grasping Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs and execution planning
• Distinguishing between RDD and DataFrame APIs and determining when to apply each
• Creating and configuring SparkSession while understanding the basics of application settings

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats: CSV, JSON, Parquet and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins and aggregations
• Implementing advanced operations such as window functions, timestamp handling and processing nested data
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Basics for Spark Users

• Understanding the relationship between Docker images and Kubernetes pods for Spark driver and executor processes – a conceptual view for cluster users rather than administrators
• Understanding what happens when a Spark job runs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings
• Understanding the distinction between settings controlled via Spark session parameters versus those managed by the platform team through cluster provisioning – and why this boundary is crucial for tuning
• Practical exercise: Examine a live Spark-on-Kubernetes job in the sandbox cluster and identify the driver and executor pods

Module 4: Spark Session Configuration on Kubernetes – In-Depth Analysis

• Understanding executor count versus executor size: trade-offs between memory and cores and how to evaluate them
• Driver and executor memory allocation, overhead, and how these translate into pod resources
• Dynamic allocation: its behaviour on Kubernetes, when it conserves resources and when it fails to do so
• Practical exercise: Execute the same job with varying executor and core configurations to compare runtime and resource usage

Module 5: Scaling Dynamics & Cost Efficiency

• Understanding how adding or removing nodes affects job behaviour, completion time and resource consumption
• Comparing many small executors against fewer large ones: performance and cost implications
• Shuffle behaviour and tuning shuffle partitions, including estimating the cost impact of configuration choices
• Practical exercise: Scale the sandbox cluster from five to ten nodes and observe the impact on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Understanding the small-files problem: why sources split into many 1–5 MB Parquet files degrade performance and disrupt partitioning
• Repartitioning and coalescing strategies
• Managing partition sizes during read and write operations
• Writing Parquet efficiently to prevent the recurrence of the small-files problem downstream
• Practical exercise: Load a dataset comprising many small Parquet partitions, apply different repartitioning strategies and compare performance before and after optimisation

Module 7: Pandas Memory Management & Failure Diagnosis

• Understanding the root causes of out-of-memory errors in Pandas and identifying the symptoms
• Applying memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when writing large datasets to CSV
• Using chunked processing and dtype optimisation for constrained environments
• Practical exercise: Reproduce a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Supplementary Tool

• Positioning Polars relative to Pandas: performance traits, lazy evaluation and memory behaviour
• Understanding where Polars fits alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Practical exercise: Refactor a Pandas-heavy transformation using Polars and compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning and memory-management principles to a realistic ETL pipeline
• Understanding optimisation considerations specific to machine-learning workloads running on the same cluster
• Applying a practical tuning workflow to systematically diagnose cost and performance issues
• Practical exercise: Complete an end-to-end mini-project combining data loading, transformation and a simple model-training step, with participants tuning the Spark configuration themselves