Overview
What was built and why it matters.
Selected applied-AI work focused on factual reliability, traceability, stakeholder confidence, human–computer interaction, and responsible data governance.
Improve trust in generated campus answers by identifying unsupported claims, weak grounding, incomplete responses, and recurring failure patterns in a live user-facing system rather than relying on anecdotal prompt testing.
Engineering approach
- Defined a repeatable prompt corpus spanning common campus intents, ambiguous questions, edge cases, and adversarial formulations.
- Created a structured evaluation rubric for factual support, relevance, completeness, consistency, source traceability, and safe handling of unanswerable requests.
- Captured and categorised failure modes so regressions could be discussed as engineering defects rather than subjective model behaviour.
- Curated an auditable campus-data layer with clearer source provenance and content organisation to strengthen grounding and reviewability.
- Repeated stress tests after data and response-handling changes to compare measured reliability against the established baseline.
Challenge
The system problem.
The system operated within a live geospatial digital-twin experience, where an answer could sound fluent while still being unsupported, incomplete, stale, or contextually misleading. Reliability work therefore needed explicit evidence, consistent scoring, and traceable source data.
Constraints
Known boundaries and deliberately unclaimed details.
- Campus information could change over time or differ across source documents, requiring source awareness and a clear distinction between unsupported, outdated, and genuinely unanswerable requests.
- Natural-language questions varied widely in specificity, phrasing, and implied context, so the evaluation corpus needed both representative and adversarial cases.
- The reported reduction had to be measured against a consistent rubric and comparable prompt set rather than inferred from a small number of favourable examples.
- Reliability improvements needed to fit around an existing live experience without depending on a complete replacement of the conversational system.
Role and ownership
Individual contribution.
- Contributed to reliability engineering for the live AI-powered conversational system.
- Owned the structure of the evaluation methodology, stress-test scenarios, failure taxonomy, and comparison process.
- Curated the auditable data layer used to improve response support and traceability.
- Analysed recurring failure patterns and used them to guide targeted changes rather than broad prompt experimentation.
- Framed results around measurable user-facing reliability and stakeholder confidence.
Contribution summary
- Designed and applied the response-evaluation and stress-testing methodology.
- Defined failure categories and repeatable checks for comparing response quality across iterations.
- Curated and organised an auditable campus-data layer to improve grounding, traceability, and stakeholder confidence.
- Connected technical evaluation results to user trust and operational governance rather than treating the system as an isolated research demonstration.
- Communicated findings in a form that supported discussion among technical, research, and stakeholder audiences.
System architecture
A profile-backed view, not an invented topology.
- A versioned prompt corpus represents common user intents, ambiguous questions, edge cases, and stress scenarios relevant to the campus environment.
- A curated campus-data layer records authoritative content with source and provenance information so generated claims can be checked against traceable evidence.
- The existing conversational system processes each prompt, while an evaluation harness captures the generated answer and associates it with the prompt version and available source evidence.
- Structured checks and reviewer criteria classify failures such as unsupported claims, factual contradictions, omission, irrelevance, ambiguity handling, and missing traceability.
- Aggregated results expose recurring failure categories and before/after comparisons, allowing reliability changes to be evaluated as a repeatable engineering process.
Key engineering decisions
Choices supported by the supplied work.
- Treat hallucination reduction as a repeatable testing and data-quality problem rather than a one-time prompt-tuning exercise.
- Separate factual support, completeness, relevance, and traceability so different failure modes do not collapse into a single subjective score.
- Use a stable baseline prompt set for before/after comparisons while continuing to add newly discovered edge cases to regression coverage.
- Improve the auditability of source data so reviewers can explain why a response passed or failed.
- Include unanswerable and ambiguous requests so the system is rewarded for appropriate uncertainty rather than confident invention.
Performance and reliability
What changed measurably.
- Reduced measured hallucinations by approximately 83% using the structured evaluation and stress-testing methodology.
- Improved the traceability of generated answers through an auditable campus-data layer and clearer source organisation.
- Converted recurring response problems into named failure categories that could be retested after changes.
- Improved stakeholder confidence by connecting quality claims to repeatable evidence rather than isolated demonstrations.
Technologies
The verified stack and focus areas.
PythonPandasApplied AIAI evaluationHallucination testingDigital twinsData governanceHuman–computer interaction
Outcome
The result without inflated claims.
A more reliable and reviewable conversational experience within a live digital-twin ecosystem used by more than 22,000 students and staff, supported by a repeatable evaluation method and an approximately 83% reduction in measured hallucinations.
Lessons and future improvements
What the evidence suggests—and what still needs verified content.
Lessons
- AI reliability improves when prompts, source data, scoring rules, and failure categories are versioned and evaluated together.
- A fluent answer is not necessarily a reliable answer; factual support, completeness, relevance, and traceability must be assessed independently.
- Evaluation becomes more useful when failure findings feed directly into data curation, product behaviour, and regression coverage.
Future improvements
- Run the evaluation suite automatically when source data, retrieval logic, prompt configuration, or model versions change, with regression thresholds that can block unsafe releases.
- Add source-freshness checks, ownership metadata, expiry rules, and conflict reporting for campus information that changes frequently.
- Expand coverage across accessibility needs, multilingual phrasing, location-dependent questions, and less-common campus services.
- Introduce production feedback capture and sampled human review so real user failures can become new regression cases.
- Add calibrated abstention and escalation behaviour for low-confidence or insufficiently supported answers.