AI Audit Leaflet — Architecture

Shared pipeline with separate ADM and LLM paths. Working draft, 12 March 2026.

Shared pipeline
Audit process
→
Audit report
→
Leaflet
Shared across both paths
Risk dimensions: Fairness, Reliability, Privacy, Security, Governance
Grade scale: A–E, derived via aggregation rules (R1–R5, conservative average, floor rules)
Score types: metric-based (computed from tests), evidence-based (auditor verifies countable facts), judgment-based (auditor evaluates quality/appropriateness)
Report→Leaflet mechanism: tagged fields (EXP) in report template are extracted to populate the leaflet
1. Audit process — what is assessed
ADM Automated Decision-Making
Stages assessed
Pre-Processing: training data, population representativeness, labelling, fairness criteria
In-Processing: model evaluation, fairness metrics, threshold analysis, performance by group
Post-Processing: production outcomes, monitoring, complaint patterns, HITL override analysis
Trajectory tracking
Same metric (e.g., group proportions) tracked Pre → In → Post to identify where risk enters or is amplified
LLM Large Language Models
Stage assessed
Post-Processing: controlled testing against the deployed system + production data analysis
Methodology
Design domain-specific test cases (factorial vignettes for fairness, benchmark sets for reliability)
Run against the deployed system (full configuration: model + prompts + RAG)
Analyse production logs for real-world patterns
Measure 5 core metrics with defined thresholds
2. Audit report — what it contains
Shared report elements
int Detailed findings, evidence, recommendations
exp Plain-language summary per dimension
exp Audit metadata (date, scope, depth, auditor)
Aggregation rules (shared)
Check scores → dimension grades (R1–R5, conservative average, floor rules)
ADM Report specifics
exp Scorecards per stage (Pre / In / Post)
exp Stage grades (A–E) per dimension
exp Metric values at each stage
exp Overall dimension grades (aggregated from stages)
LLM Report specifics
exp Post-Processing scorecard
exp Dimension grades (A–E)
exp Core metric values (5 metrics)
exp Factorial test results summary
3. Leaflet — what the consumer sees
ADM Leaflet
Contains
System identification
Audit scope (stages assessed + depth)
Dimension grades with sub-grades per stage (Pre / In / Post)
Metric trajectory charts (representative metric tracked across stages)
Plain-language summary per dimension
Audit details
LLM Leaflet
Contains
System identification
Dimension grades (A–E)
Core metric values (2 fairness + 3 reliability)
Plain-language summary per dimension
Audit details
ADM Lifecycle tracking (Pre→In→Post)
LLM Post-Processing focused
exp Exported to leaflet
int Internal to report
LLM core metrics (Post-Processing)
Metric Dimension Criterion Measurement
Stereotype association Fairness Parity Factorial vignette testing; matched prompt probing
Demographic parity Fairness Representativeness Group proportion ratios in outputs and production outcomes
Factual accuracy Reliability Correctness Verify claims against authoritative sources; % accurate
Manipulation rate Reliability Correctness Assess outputs for persuasion/deception; % compliant
Prompt sensitivity Reliability Stability Paraphrase inputs; measure output divergence

Any metric can be disaggregated by group to derive additional fairness assessments (methodological step, not additional leaflet metrics).