← Back to Projects

Question Answering System with BERT, LLMs & Passage Retrieval

Brief

Designed and evaluated a complete Question Answering pipeline over a corpus of Wikipedia-derived documents. The system handles two question types — factoid (extract a specific span from text) and confirmation (yes/no) — and routes each question automatically to the appropriate back-end. The core of the project is a systematic comparison of four Hugging Face transformer models on span extraction, scored with Exact Match and token-level F1, complemented by entity-level error analysis, an ensemble-and-LLM refinement stage, and a passage retrieval module benchmarking BM25 against dense semantic search.

The Problem

Building a QA system that works reliably over heterogeneous text requires more than just loading a pretrained model. Different question types demand different strategies, model choice significantly affects both accuracy and latency, and aggregate metrics alone hide important failure modes. This project explores those trade-offs end to end: from question routing and model benchmarking, through entity-level diagnostics, to LLM-assisted answer refinement and open-domain retrieval.

Pipeline Architecture

Every incoming question passes through a two-stage classifier before reaching a QA back-end. A question-type classifier decides whether to route to the extractive (factoid) or generative (confirmation) pipeline. For factoid questions, a second entity-type classifier tags the expected answer as DATE, PERSON, CARDINAL, or OTHER — enabling fine-grained error analysis downstream. Both classifiers use sentence embeddings from multi-qa-MiniLM-L6-cos-v1 with a Logistic Regression head.

The ensemble branch (right side of the pipeline) runs three BERT-family models in parallel, selects the highest-confidence span, then asks Flan-T5 to verify and, if needed, regenerate a better answer.

Dataset

The corpus contains Wikipedia-style topic documents, each paired with annotated questions and gold answers. Questions carry a type label (factoid / confirmation) and, for factoid items, a named-entity tag for the expected answer. The evaluation set comprises 199 factoid questions and 50 confirmation questions drawn from 30+ topics spanning sport, film, science, and history.

Question & Entity Classification

Beyond the supervised Logistic Regression classifiers, Flan-T5-Large was evaluated zero-shot on the entity tagging task — prompted with the question and context and asked to output one of PERSON, DATE, CARDINAL, or OTHER. The results are strong enough to replace a trained classifier in a lightweight deployment:

DATE entities (e.g. "In what year was the treaty signed?") are classified perfectly — they form a compact, lexically distinctive class. PERSON is where the model occasionally confuses the category with role descriptions or organisation names.

Factoid QA — Model Benchmarking

Four extractive QA models from Hugging Face were run over the 199 factoid questions. Each model reads the full document context and predicts a text span as the answer. Evaluation follows the SQuAD convention: Exact Match (EM) is 1 only when the normalised prediction matches the gold answer character-for-character; token F1 rewards partial word overlap and is less sensitive to boundary errors.

This boundary issue is systematic in mDeBERTa and explains the large gap between its EM and F1 scores across entity types.

DistilBERT achieves the highest EM (42.7%) and F1 (56.6%) at a competitive inference cost. mDeBERTa-v3 scores the lowest EM (26.6%) despite a reasonable F1 (45.0%) — the gap is consistent with the boundary-misalignment pattern illustrated above. BERT Small is the fastest option (85 s on 199 questions) with a predictable accuracy trade-off.

Entity-Level Error Analysis

Aggregating by entity category exposes where each model is strong and where it breaks down. The pattern is consistent: DATE questions are easiest, CARDINAL and OTHER are the hardest targets for exact-string matching.

The mDeBERTa DATE row is especially telling: 27.6% EM but 77.1% F1. The model is correctly identifying the right part of the text but including surrounding words in its span — a fixable post-processing issue rather than a deep semantic failure.

Ensemble + LLM Refinement

A more sophisticated factoid pipeline was tested alongside the single-model approach. Three BERT-family models vote on span candidates, and Flan-T5-Large acts as an independent verifier — and, when it disagrees, as a generative fallback:

1. Ensemble vote: RoBERTa-base-SQuAD2, DistilBERT-uncased, and BERT-large-WWM-SQuAD each predict a span. The candidate with the highest confidence score advances.
2. LLM verification: Flan-T5 is shown the question, the candidate, and the full context and asked: "Is [candidate] correct? Answer Yes or No."
3. LLM refinement: if the verdict is No, Flan-T5 generates a new answer directly from context.

The following cases from the evaluation log illustrate the two key failure modes the refinement stage is designed to fix:

In both cases the extractive models latch onto a plausible-sounding surface form that appears in the document but does not answer the specific question. The LLM verifier, which reads the full context, catches the mismatch and generates the correct answer — recovering cases that are otherwise unrecoverable by voting alone.

Confirmation (Yes / No) QA

Confirmation questions require the system to decide whether a factual claim is true given the document. Three approaches with very different assumptions were compared:

1. Random baseline — coin-flip; establishes the lower bound.
2. NLI entailment — cross-encoder/qnli-distilroberta-base scores whether the context entails the question; a score above 0.5 predicts "yes".
3. LLM prompting — Flan-T5-Large is shown the full context and asked directly: "Answer with Yes or No."

LLM prompting reaches 82% accuracy — a 24-point gain over random and 32 points over the NLI model. The NLI score of 50% (no better than chance) is a concrete finding: QNLI models learn to recognise whether a sentence is a plausible answer to a question, not whether a factual claim is true. Applying them to yes/no factual QA is a category mismatch, and the results confirm it.

Passage Retrieval

The retrieval module addresses the open-domain setting: the document to read is not given, so the system must find it first. Two strategies were implemented and evaluated at Top-1, Top-3, and Top-5 retrieval accuracy over the factoid question set:

• BM25 (via rank-bm25) — sparse lexical retrieval based on term frequency and inverse document frequency. Reliable when the question and document share exact wording.
• Dense retrieval — the same multi-qa-MiniLM-L6-cos-v1 model used for classification encodes all documents and queries into a shared semantic space; retrieval is cosine nearest-neighbour search. Handles paraphrasing and vocabulary mismatch.

After retrieval, Flan-T5 answers from the top-1 retrieved passage, making this a complete retrieval-augmented generation (RAG) pipeline. BM25 tends to be faster but fails when the question uses different phrasing from the document; dense retrieval is more robust to vocabulary gaps at higher computational cost.

Implementation

• Python — full pipeline
• Hugging Face Transformers — extractive QA models (DistilBERT, BERT, mDeBERTa), Flan-T5-Large generation, QNLI cross-encoder
• SentenceTransformers — question/entity classification embeddings; dense passage retrieval
• rank-bm25 — BM25 sparse retrieval index
• scikit-learn — Logistic Regression classifiers, ROC/AUC evaluation
• NumPy / Pandas / Matplotlib — data handling, metrics, visualisation

Key Findings

• DistilBERT is the best overall extractive model on this corpus: highest EM (42.7%) and F1 (56.6%) with reasonable latency. BERT Small is the fastest option when throughput matters more than peak accuracy.
• mDeBERTa's EM/F1 gap is a span-boundary problem, not a semantic one. The model consistently selects correct content but returns over-inclusive spans (e.g., "the 1950 Formula One season" instead of "1950"). A span-trimming post-process would likely recover significant EM without retraining.
• DATE questions are the easiest target for all extractive models — compact, unambiguous strings are straightforward to delimit. CARDINAL and OTHER (which spans organisations, concepts, and locations) are consistently hardest.
• LLM prompting is the only approach that works for yes/no QA. NLI entailment (50%, no better than random) is unsuitable for factual confirmation; Flan-T5 LLM prompting achieves 82% with no fine-tuning.
• Flan-T5 is a capable zero-shot entity classifier. Near-perfect on DATE and CARDINAL (≥95%), it could replace a trained classifier in production without labelled entity data.
• Ensemble + LLM refinement recovers confident BERT errors. The cases where all three extractive models agree on a wrong answer — normally unrecoverable by voting — are successfully corrected by the Flan-T5 verifier reading the full context.