How each pipeline fails

Failure types per pipeline, assigned by a deterministic classifier (and the faithfulness judge). Click through to see the full answer and trace.

RAG runGraphRAG runAgentic GraphRAG run

Small sample (rag: n=5, graphrag: n=5, agent: n=5). Confidence intervals are wide, so treat differences as indicative, not conclusive.

Failure types

Number of questions each pipeline got wrong, by cause.

Every question a pipeline did not fully answer correctly.

SearchCategoryPipelineFailure

Showing 7 of 7 failures.

Question	Category	Pipeline	Failure	Answered
Who is the statutory auditor of Tata Steel in FY2023-24? Q-SF-0002	Single fact	RAG	Wrong abstention	not found in the data	Inspect
Who is the statutory auditor of Tata Steel in FY2023-24? Q-SF-0002	Single fact	GraphRAG	Wrong abstention	not found in the data	Inspect
Who is the statutory auditor of Tata Steel in FY2023-24? Q-SF-0002	Single fact	Agentic GraphRAG	Wrong abstention	not found in the data	Inspect
Who is the Executive Director of Tata Motors in FY2023-24? Q-SF-0008	Single fact	RAG	Hallucination	Mr. Girish Wagh	Inspect
Who is the Executive Director of Tata Motors in FY2023-24? Q-SF-0008	Single fact	GraphRAG	Hallucination	Mr. Girish Wagh	Inspect
Who is the Executive Director of Tata Motors in FY2023-24? Q-SF-0008	Single fact	Agentic GraphRAG	Bad query	(error)	Inspect
Why did Sanjiv Bajaj resign as Chairman of Bajaj Finance in FY2023-24? Q-UNA-0004	Unanswerable	Agentic GraphRAG	Bad query	(error)	Inspect