IndQA – Benchmark for Indian Cultural and Linguistic Understanding

IndQA is a newly released benchmark that assesses how well AI systems comprehend and reason about Indian cultural contexts in native languages. It contains 2,278 expertly crafted questions spanning 12 languages and ten cultural domains, providing a rigorous, rubric‑based evaluation framework for multilingual large language models.

Design and Methodology

The benchmark was built through a multi‑stage process that prioritizes cultural authenticity and linguistic diversity. Expert domain specialists authored each prompt in the target language, supplied English translations for auditability, and defined detailed grading criteria to ensure consistent scoring.

Question Construction and Expert Involvement

261 native‑level experts from fields such as literature, architecture, and culinary arts drafted reasoning‑heavy prompts. Each question reflects real‑world cultural nuances, and a peer‑review loop refined the items until expert sign‑off, guaranteeing domain relevance and linguistic fidelity.

Rubric‑Based Grading System

For every question, a rubric lists specific criteria with weighted points. An automated grader checks model responses against these criteria, aggregating points to produce a final score that mirrors human essay grading standards.

Comparative Evaluation and Findings

IndQA was used to track performance trends of frontier models, revealing measurable gains in Indian language handling while highlighting persistent gaps. The benchmark’s adversarial filtering—excluding questions that top OpenAI models answered correctly—ensures headroom for future improvements.

Performance Across Models

Evaluations show that newer models outperform earlier versions on many domains, yet scores remain modest in areas like legal reasoning and regional folklore, indicating targeted research opportunities.

Limitations and Caveats

Because question sets differ across languages, IndQA does not serve as a direct language leaderboard. Its adversarial design may bias results toward OpenAI models, and cross‑language comparisons should be interpreted cautiously.

For broader context on the capabilities of generative artificial intelligence and the role of large language models in multilingual evaluation, see the linked resources.