I led a four-person team building an AI knowledge assistant for the Royal Audit Authority. The project connects natural-language questions to approved Acts, regulations, manuals, and policies, helping auditors find relevant information and check the original sources behind an answer.
- Client & programme
- Royal Audit Authority, Bhutan
- My role
- Team lead, full-stack & AI/ML development
- Collaboration
- Four-person project team
- Project status
- Developed & evaluated
The problem
Institutional knowledge was difficult to find and reuse.
RAA auditors work with Acts, rules, circulars, manuals, policies, and procedural guidance. The project makes that approved material easier to search and reuse through a consistent institutional reference.
The project focused on a conversational route into approved documents. An auditor could ask a question in ordinary language, read an explanation grounded in retrieved material, and return to the original document to check the evidence. The assistant supports that research process; professional audit decisions remain with the auditor.
My contribution
Connecting the application and the retrieval pipeline.
My role combined team leadership, full-stack development, and AI/ML work. I worked across the React and TypeScript interface, FastAPI backend, document ingestion, semantic search, and the RAG pipeline that connects retrieved evidence to generated answers.
My contribution included document preparation, semantic retrieval, reranking, and source-grounded answer generation. I also worked on role-based access, citation tracking, and evaluation of the retrieval and generation pipeline using RAGAS.
The system was developed collaboratively by a four-person team. My contribution spans the application and AI integration; the finished project also reflects shared work on requirements, design, development, testing, and documentation.
The user experience
Ask a question, then inspect the source.
The user-facing experience centres on a chat workspace with follow-up questions, conversation history, and source references. Citations connect an answer to its document and relevant section or page, giving the auditor a route to examine the underlying text rather than relying on the generated explanation alone.
The knowledge base also needs people responsible for its content and operation. The project separates conversational access, document curation, and system administration into three roles.
- Auditors
- Ask questions, review cited material, and continue a conversation without administrative access to change the knowledge base.
- Knowledge management administrators
- Upload approved PDF and DOCX documents and maintain metadata such as titles, categories, and effective dates, alongside access to the assistant.
- System administrators
- Manage users and permissions, review audit records, and monitor application and AI activity.
The RAG workflow
Prepare the documents before generating the answer.
The system has two connected paths: preparing approved documents for search, and answering a question against that indexed material. Keeping those paths separate lets document processing run in the background while the interactive application handles queries.
Ingest and extract
An authorised administrator uploads a PDF or DOCX file. Background processing extracts the text and retains the document metadata needed to identify the source later.
Split with context
Documents are divided into searchable passages while preserving enough surrounding context to support useful retrieval and source references.
Embed and index
Passages are prepared for semantic search and remain linked to the original document so retrieved evidence can be identified and inspected.
Retrieve and rerank
The question is used to retrieve candidate passages, which are ranked for relevance before selected evidence is used to prepare an answer.
Generate and cite
The assistant produces an explanation from the retrieved context, with references that connect the answer back to its source material.
Handle missing evidence
The intended response when the repository does not support a question is to say that no verified information is available, rather than supply a speculative answer.
Engineering decisions
Treat retrieval quality and traceability as product features.
Document structure influenced the ingestion strategy. Dense reference material and narrative documents were evaluated separately to preserve useful context. This makes document preparation part of the answer-quality work.
The application combines a React and TypeScript interface with a FastAPI backend and PostgreSQL. Document preparation runs separately from interactive queries, helping the user experience stay focused on asking questions and reviewing evidence.
Access controls, approved document curation, source references, and evaluation support the assistant's reliability. An answer remains connected to material the auditor can inspect before using it in their work.
Evaluation
Measure the generated answer, not just the search result.
The project evaluated the pipeline with RAGAS using 20 domain-specific auditing questions. The experiments compared dense and hybrid retrieval, document-splitting strategies, and a baseline RAG pipeline against a reranked configuration.
In the baseline-versus-reranked experiment, the Advanced RAG configuration recorded faithfulness of 0.90 and answer relevancy of 0.89. Both improved over the baseline in that comparison, while context precision decreased. That trade-off helped focus the evaluation on the final answer as well as the passages retrieved.
These are results from the project’s evaluation set. They provide evidence for the selected pipeline, rather than a general accuracy percentage or a guarantee for every question. Keeping the sources inspectable and continuing evaluation as the document collection changes remain important parts of the approach.
Outcome & next steps
A working assistant, with institutional rollout treated separately.
The project brought document ingestion, conversational search, cited answers, role-based administration, and monitoring into one application. The final report documents functional, integration, browser, and device checks alongside the evaluation work.
The implementation reached a demonstration deployment. Institutional rollout and operational acceptance are separate from the reported evaluation results.
RAA recognised my contribution to KMS with a Certificate of Appreciation in July 2026. The main technical lesson was that a useful knowledge assistant depends on the whole evidence chain: well-prepared documents, relevant retrieval, controlled generation, and an accessible route back to the source.