Text2SQL — Fine-tuning a ≤3B LLM for SQL Generation

End-to-end pipeline for fine-tuning a small (≤3B) open-source LLM to convert plain-English questions into executable SQL. Fine-tuned Qwen2.5-Coder-1.5B with QLoRA via Unsloth on the BIRD benchmark, evaluated by execution accuracy on real SQLite databases. Lifted the valid-SQL rate from 40% → 73.5% and execution accuracy from 14.0% → 15.5% on 200 BIRD-dev questions — all on a free Kaggle T4 GPU. Includes dataset exploration, preprocessing, SFT training, inference, and execution-based evaluation notebooks, plus a written report and 3 ablation experiments.

Proof points

• 40% → 73.5% Valid-SQL
• 200 BIRD-dev Questions
• Free T4 GPU

Technologies

Python, QLoRA, Unsloth, TRL, HuggingFace, BIRD, Qwen2.5-Coder, Google Colab, SQLite

Links

• Source repository

Problem

Non-technical users need to query databases in plain English, but small open-source LLMs (≤3B params) produce malformed or non-executable SQL out of the box — only 40% valid SQL at baseline.

Approach

Built a complete fine-tuning pipeline using QLoRA + Unsloth for memory-efficient SFT on the BIRD benchmark. Applied completion-only loss masking with TRL SFTTrainer, serialized SQLite schemas for prompt construction, and evaluated using execution accuracy (comparing gold vs. predicted result sets on real databases). Designed 3 ablation experiments: baseline (zero-shot 1.5B), exp1 (1.5B + BIRD QLoRA), exp2 (0.5B ablation), and exp3 (BIRD + SynSQL cross-domain). Entire training pipeline fits on a free Kaggle T4 GPU (16 GB VRAM).

Results

Fine-tuning nearly doubled the valid-SQL rate (40% → 73.5%) and improved execution accuracy from 14.0% to 15.5% on 200 BIRD-dev questions. Adapter published to HuggingFace (Shiverion/qwen2.5-coder-1.5b-bird-qlora). Complete pipeline runs on free compute with full evaluation framework and written report.