Pandas vs Polars vs DuckDB Benchmark
A comprehensive benchmark of Pandas, Polars, and DuckDB on a 16GB laptop processing 13.1 million rows (Susenas 2024). Demonstrated 98% memory reduction and 100x speedup using modern libraries. Includes detailed performance analysis across filtering, aggregation, and join operations.
Proof points
- 13.1M Rows
- 98% Less RAM
- 100x Faster
Technologies
Python, Pandas, Polars, DuckDB, Jupyter Notebook
Links
Problem
Traditional data analysis with Pandas on 13.1 million rows caused memory crashes on a standard 16GB laptop due to high memory overhead.
Approach
Benchmarked Pandas against modern alternatives (Polars, DuckDB) using 3 scenarios: Simple Filtering, Window Functions, and Advanced Joins. Tested on both local hardware (16GB RAM) and cloud (47GB RAM).
Results
DuckDB and Polars proved vastly superior. Polars achieved 98% memory reduction (0.42GB vs 41GB) and DuckDB ran complex joins in 10s where Pandas crashed. Established a decision framework for choosing the right tool based on hardware constraints.