YCG — ISO Compliance Document Auditor

What It Is#

At Devsinc, I worked on an internal AI document auditing tool called YCG. The platform helps evaluate company Quality Manuals and standard operating procedures against formal regulatory standards—specifically ISO 13485:2016 (the international quality standard for medical devices).

Quality Manuals are often hundreds of pages long and contain dense, cross-referenced policies. Reviewing them manually clause-by-clause takes days of auditor time. YCG automates the first-pass audit by checking uploaded documents against a structured ISO standards registry, generating a clause-level compliance report and highlighting any missing requirements.

Why Single-Prompt LLM Auditing Fails#

When we first tested compliance checks with LLMs, sending an entire 200-page manual in a single prompt led to useless answers:

• The model produced broad, polite summaries like "The manual largely complies with ISO guidelines", completely missing specific sub-clause omissions (e.g., Section 7.3 Design and Development controls).
• Generic prompts hallucinated compliance on missing sections because the overall tone of the document sounded professional.

To make the audit genuinely useful, we had to break the evaluation down into discrete steps.

Architecture & How It Works#

I structured YCG as a decoupled system: a FastAPI backend that handles the multi-stage Claude API pipeline and document parsing, and a Streamlit frontend where reviewers upload files and inspect gap reports.

Quality Manual (PDF / DOCX)
             │
             ▼
┌──────────────────────────────┐
│  Streamlit Review Dashboard  │ (Upload, Session Status & Job Browser)
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│       FastAPI Backend        │ (Endpoints in backend/main.py)
└──────────────┬───────────────┘
               │
       ┌───────┴───────┐
       ▼               ▼
┌──────────────┐ ┌──────────────┐
│ ISO Registry │ │  Claude API  │
│ JSON Files   │ │  Multi-Stage │
└──────┬───────┘ └──────┬───────┘
       └───────┬────────┘
               ▼
┌──────────────────────────────┐
│  Structured Audit Report     │ (Clause scores, evidence & gap fixes)
└──────────────────────────────┘

1. External ISO Standards Registry (iso_registry.py)

Instead of hardcoding ISO clauses into prompts, we modeled standards as structured JSON files in data/test JSON/. Each clause lists its standard identifier, required evidence criteria, and guidance notes. This separation meant we could support other standards (like ISO 9001) without modifying the backend evaluation code.

2. Multi-Stage Evaluation Pipeline (prompts.py)

Rather than asking Claude to judge the entire manual at once, the backend runs a two-stage prompt pipeline:

• Stage 1 (PIPELINE_PROMPT_1): Scans the uploaded text to extract and group sections related to a specific ISO clause.
• Stage 2 (PIPELINE_PROMPT_2): Evaluates the extracted policy text strictly against the requirement manifest, returning structured JSON with compliance status (Compliant, Partially Compliant, or Non-Compliant), verbatim quotes as evidence, and suggested remediations.

3. Streamlit Review Interface

The Streamlit frontend (frontend/app.py and views/job_details.py) gives internal reviewers a clean dashboard:

• Upload documents and select the target ISO standard.
• Monitor active audit sessions.
• Browse clause-by-clause scores with expandable sections showing exactly where the manual met or missed the requirement.

Takeaways#

• Multi-stage prompts are required for audits: Forcing the model to first locate relevant text before evaluating it against a checklist eliminates hallucinations and produces clear, actionable feedback.
• Decoupling standards from code keeps tools maintainable: Storing standard clauses in external JSON manifests makes adding or updating regulatory frameworks straightforward.