VerdictBench: Benchmarking Long Context and Retrieval-Augmented Generation on Indonesian Constitutional Court Verdicts
An empirical benchmark comparing Long Context, Dense RAG, and Multi-Stage RAG on 50 Indonesian Constitutional Court verdicts and 300 human-reviewed QA pairs. Phase 2 shows Long Context and Dense RAG are statistically tied on gold-evidence faithfulness across Gemini 2.5 Flash and GPT-4o Mini, while Dense RAG is 16-25x cheaper and avoids the long-verdict non-response failures seen with Long Context. Multi-Stage RAG underperforms both baseline methods under oracle gold-evidence evaluation, making the study a cautionary result for adding query rewriting, hybrid search, metadata filters, and reranking without domain validation.
Proof points
- LC ~= Dense RAG
- 24.8x Cheaper
- 300 QA Pairs
Technologies
Python, Gemini 2.5 Flash, GPT-4o Mini, FAISS, RAG, BM25, IndoBERT, NLP
Links
Problem
The rapid expansion of LLM context windows raises an urgent question: if an entire legal document fits into the prompt, is RAG still necessary? No systematic benchmark existed for Indonesian legal text to answer this.
Approach
Built a complete research pipeline: corpus construction from 887 raw MK verdicts → stratified sampling of 50 → 300 human-reviewed QA pairs → controlled experiments across 3 architectures × 2 models. Evaluated with a two-model faithfulness judge, BERTScore (IndoBERT), ablation study, length-sensitivity analysis, and needle-in-a-haystack evaluation.
Results
Phase 2 shows Long Context and Dense RAG are statistically tied on gold-evidence faithfulness across Gemini 2.5 Flash and GPT-4o Mini. Dense RAG remains 24.8x cheaper under Gemini Flash and 16.3x cheaper under GPT-4o Mini, while avoiding Long Context's 51 long-verdict quota failures in Phase 1. Baseline Dense RAG also topped the ablation study at 0.604 gold-evidence faithfulness; every Multi-Stage RAG augmentation reduced oracle grounding.
VerdictBench: Research Paper Summary
1. Executive Summary
This study provides an empirical comparison of Long Context (LC), Dense RAG, and Multi-Stage RAG architectures on Indonesian Constitutional Court verdicts. The main result is practical: LC and Dense RAG are statistically tied on Phase 2 gold-evidence faithfulness, while Dense RAG remains much cheaper and more operationally reliable on long verdicts. Multi-Stage RAG underperforms both, showing that more retrieval stages do not automatically improve legal QA.
2. Research Questions
- RQ1 — Does Long Context outperform RAG when the entire document fits within the context window?
- RQ2 — How does verdict length affect answer quality and response coverage?
- RQ3 — Which Multi-Stage RAG components improve or degrade faithfulness?
- RQ4 — How does the choice of LLM interact with architectural choice?
3. Methodology
- Corpus: 50 Indonesian Constitutional Court (MK) verdicts, stratified by length (15 short, 20 medium, 15 long)
- QA Dataset: 300 human-reviewed question-answer pairs across 4 cognitive types
- Models: Gemini 2.5 Flash (1M context) and GPT-4o Mini (128K context)
- Evaluation: Gold-evidence faithfulness, BERTScore, cost, latency, ablation analysis, length sensitivity, NIAH, and human legal accuracy spot-check
4. Key Findings
Architecture Finding: LC ~= Dense RAG > Multi-Stage RAG
- LC and Dense RAG have negligible Phase 2 differences
- Gemini 2.5 Flash: dz = 0.008, p = 0.43 for LC vs Dense RAG
- GPT-4o Mini: dz = 0.017, p = 0.49 for LC vs Dense RAG
- Both LC and Dense RAG outperform Multi-Stage RAG
Cost Efficiency
- Dense RAG is 24.8x cheaper than Long Context under Gemini Flash
- Dense RAG is 16.3x cheaper than Long Context under GPT-4o Mini
- The practical Dense RAG advantage is cost and reliability, not superior faithfulness
Ablation: Component-Level Attribution
- Baseline Dense RAG: 0.604 gold-evidence faithfulness, highest ablation condition
- Hybrid BM25+dense search: 0.585, least harmful augmentation
- Query rewriting: 0.453, most harmful augmentation
- Full Multi-Stage RAG: 0.472, below the Dense RAG baseline
Length Sensitivity
- LC faithfulness quality increases with verdict length among answered queries
- Long-verdict LC answered-only faithfulness: 0.628
- Long-verdict LC coverage-adjusted faithfulness: 0.272 due to 51/90 quota-exhaustion failures
Needle-in-a-Haystack
- Threshold success: LC 0.400, Multi-Stage RAG 0.333, Dense RAG 0.300
- The NIAH result does not support retrieval clearly beating LC on deep evidence
- A separate retrieval-success metric is needed to test the lost-in-the-middle hypothesis directly
5. Tech Stack
- Python, FAISS, BM25, Google text-embedding-004, IndoBERT
- Two-model judge: Gemini 3 Flash Preview + Gemini 2.0 Flash
- Statistical: paired Wilcoxon signed-rank, bootstrap CIs, paired dz
6. Impact
First benchmark for Indonesian legal NLP. Publicly available corpus, QA dataset, and pipeline at github.com/Shiverion/VerdictBench-LCvsRAG.