QA for RAG Chatbot: When Source Attribution Can’t Be Wrong

Testing an enterprise AI chatbot with document retrieval, source attribution, and SSO integration. From 800-page PDFs to permission escalation risks — here’s what QA caught.

What we did:

About project and client
Before & after
Project duration
Challenge
Solution
Technologies
Types of testing
Results

About project

Solution

Manual testing, Integration testing, Security testing, Regression testing

Technologies

Python (FastAPI), React (Next.js), Google Cloud, Document AI, Microsoft Entra ID

Country

United States

Industry

eCommerce

Client

A US-based company building an AI-powered knowledge management system for enterprise use. The chatbot processes internal documents and provides answers with source references. The system required enterprise-grade security with SSO integration and role-based permissions.

Project overview

Let us make sure your product is 100% release- and compliance-ready.

Let’s talk!

Before

No unified system for querying internal documents
Manual search through large document libraries
No role-based access control for sensitive data

After

AI chatbot tested for accuracy and source attribution
Document processing pipeline verified for files up to 800+ pages
SSO and permission system validated, including edge case risks

Project Duration

4 months

Team Composition

Testing & QA – 2

Challenge

The system had to process various document formats, extract text, split it into chunks, and make it searchable through natural language queries.

QA needed to verify:

Role-based permissions don’t create security gaps
Document pipeline handles large files (800+ pages) without breaking
Chatbot responses match actual document content
Source attribution points to correct files
SSO authentication works with Microsoft Entra ID

Solutions

Document Pipeline Testing

We verified each processing stage: upload to bucket, text extraction, chunking, and embedding creation. Tested edge cases including 800-page PDFs and various file formats.

Chatbot Accuracy Testing

Validated that responses match document content. Verified source attribution points to correct files — not hallucinated sources.

SSO and Security Testing

Tested authentication flows via Microsoft Entra ID, session management, and verified that roles are created inside the web app, not assigned according to Entra ID role.

Permission Edge Cases

Identified a critical risk: lower-level roles with “assign role” permission could reassign admin privileges. Documented and reported to client with fix recommendations.

Regression Testing

Reran test suites after each update to catch new bugs and maintain system stability.

Technologies

We picked a toolset that could support the full connected-device testing workflow, from test management and cross-platform mobile coverage to sensor data validation.