VerdictBench: Benchmarking Long Context and Retrieval-Augmented Generation on Indonesian Constitutional Court Verdicts

An empirical benchmark comparing Long Context, Dense RAG, and Multi-Stage RAG on 50 Indonesian Constitutional Court verdicts and 300 human-reviewed QA pairs. Phase 2 shows Long Context and Dense RAG are statistically tied on gold-evidence faithfulness across Gemini 2.5 Flash and GPT-4o Mini, while Dense RAG is 16-25x cheaper and avoids the long-verdict non-response failures seen with Long Context. Multi-Stage RAG underperforms both baseline methods under oracle gold-evidence evaluation, making the study a cautionary result for adding query rewriting, hybrid search, metadata filters, and reranking without domain validation.

Proof points

  • LC ~= Dense RAG
  • 24.8x Cheaper
  • 300 QA Pairs

Technologies

Python, Gemini 2.5 Flash, GPT-4o Mini, FAISS, RAG, BM25, IndoBERT, NLP

Links

Problem

The rapid expansion of LLM context windows raises an urgent question: if an entire legal document fits into the prompt, is RAG still necessary? No systematic benchmark existed for Indonesian legal text to answer this.

Approach

Built a complete research pipeline: corpus construction from 887 raw MK verdicts → stratified sampling of 50 → 300 human-reviewed QA pairs → controlled experiments across 3 architectures × 2 models. Evaluated with a two-model faithfulness judge, BERTScore (IndoBERT), ablation study, length-sensitivity analysis, and needle-in-a-haystack evaluation.

Results

Phase 2 shows Long Context and Dense RAG are statistically tied on gold-evidence faithfulness across Gemini 2.5 Flash and GPT-4o Mini. Dense RAG remains 24.8x cheaper under Gemini Flash and 16.3x cheaper under GPT-4o Mini, while avoiding Long Context's 51 long-verdict quota failures in Phase 1. Baseline Dense RAG also topped the ablation study at 0.604 gold-evidence faithfulness; every Multi-Stage RAG augmentation reduced oracle grounding.

VerdictBench: Research Paper Summary

1. Executive Summary

This study provides an empirical comparison of Long Context (LC), Dense RAG, and Multi-Stage RAG architectures on Indonesian Constitutional Court verdicts. The main result is practical: LC and Dense RAG are statistically tied on Phase 2 gold-evidence faithfulness, while Dense RAG remains much cheaper and more operationally reliable on long verdicts. Multi-Stage RAG underperforms both, showing that more retrieval stages do not automatically improve legal QA.

2. Research Questions

  1. RQ1 — Does Long Context outperform RAG when the entire document fits within the context window?
  2. RQ2 — How does verdict length affect answer quality and response coverage?
  3. RQ3 — Which Multi-Stage RAG components improve or degrade faithfulness?
  4. RQ4 — How does the choice of LLM interact with architectural choice?

3. Methodology

  • Corpus: 50 Indonesian Constitutional Court (MK) verdicts, stratified by length (15 short, 20 medium, 15 long)
  • QA Dataset: 300 human-reviewed question-answer pairs across 4 cognitive types
  • Models: Gemini 2.5 Flash (1M context) and GPT-4o Mini (128K context)
  • Evaluation: Gold-evidence faithfulness, BERTScore, cost, latency, ablation analysis, length sensitivity, NIAH, and human legal accuracy spot-check

4. Key Findings

Architecture Finding: LC ~= Dense RAG > Multi-Stage RAG

  • LC and Dense RAG have negligible Phase 2 differences
  • Gemini 2.5 Flash: dz = 0.008, p = 0.43 for LC vs Dense RAG
  • GPT-4o Mini: dz = 0.017, p = 0.49 for LC vs Dense RAG
  • Both LC and Dense RAG outperform Multi-Stage RAG

Cost Efficiency

  • Dense RAG is 24.8x cheaper than Long Context under Gemini Flash
  • Dense RAG is 16.3x cheaper than Long Context under GPT-4o Mini
  • The practical Dense RAG advantage is cost and reliability, not superior faithfulness

Ablation: Component-Level Attribution

  • Baseline Dense RAG: 0.604 gold-evidence faithfulness, highest ablation condition
  • Hybrid BM25+dense search: 0.585, least harmful augmentation
  • Query rewriting: 0.453, most harmful augmentation
  • Full Multi-Stage RAG: 0.472, below the Dense RAG baseline

Length Sensitivity

  • LC faithfulness quality increases with verdict length among answered queries
  • Long-verdict LC answered-only faithfulness: 0.628
  • Long-verdict LC coverage-adjusted faithfulness: 0.272 due to 51/90 quota-exhaustion failures

Needle-in-a-Haystack

  • Threshold success: LC 0.400, Multi-Stage RAG 0.333, Dense RAG 0.300
  • The NIAH result does not support retrieval clearly beating LC on deep evidence
  • A separate retrieval-success metric is needed to test the lost-in-the-middle hypothesis directly

5. Tech Stack

  • Python, FAISS, BM25, Google text-embedding-004, IndoBERT
  • Two-model judge: Gemini 3 Flash Preview + Gemini 2.0 Flash
  • Statistical: paired Wilcoxon signed-rank, bootstrap CIs, paired dz

6. Impact

First benchmark for Indonesian legal NLP. Publicly available corpus, QA dataset, and pipeline at github.com/Shiverion/VerdictBench-LCvsRAG.