Pandas vs Polars vs DuckDB Benchmark

A comprehensive benchmark of Pandas, Polars, and DuckDB on a 16GB laptop processing 13.1 million rows (Susenas 2024). Demonstrated 98% memory reduction and 100x speedup using modern libraries. Includes detailed performance analysis across filtering, aggregation, and join operations.

Proof points

  • 13.1M Rows
  • 98% Less RAM
  • 100x Faster

Technologies

Python, Pandas, Polars, DuckDB, Jupyter Notebook

Links

Problem

Traditional data analysis with Pandas on 13.1 million rows caused memory crashes on a standard 16GB laptop due to high memory overhead.

Approach

Benchmarked Pandas against modern alternatives (Polars, DuckDB) using 3 scenarios: Simple Filtering, Window Functions, and Advanced Joins. Tested on both local hardware (16GB RAM) and cloud (47GB RAM).

Results

DuckDB and Polars proved vastly superior. Polars achieved 98% memory reduction (0.42GB vs 41GB) and DuckDB ran complex joins in 10s where Pandas crashed. Established a decision framework for choosing the right tool based on hardware constraints.