What it does
Tests AI/ML-based features against ISO/IEC 25010 plus AI-specific quality characteristics: classifies the model and picks a non-exact-match oracle, sets measurable ML metric thresholds, validates the data pipeline (no PII), measures bias/fairness and robustness, checks explainability, and plans drift monitoring, then gates on the thresholds. Flags where human/ethical sign-off is required.
When to use it
When the system under test includes AI/ML features.
Prerequisites
qa.config.yml with paths, risk_areas, gates thresholds, test_data.strategy/sensitive_data_rule; a model/feature to test; synthetic data only (no real PII).
Output
AI test approach and case specification (ISO/IEC/IEEE 29119-3 Test Design / Test Case Specification) -> docs_dir/AI-TEST-<feature>.md, with automatable checks under tests_dir and failing results routed to /qa:triage.
Mechanics
How it works
- Read config; resolve the AI/ML feature (stop if no AI/ML component)
- Classify the model and pick a non-exact-match oracle strategy per test
- Set ML functional-performance metrics with numeric acceptance thresholds
- Validate data (split integrity, representativeness, skew) honoring the PII rule
- Measure bias/fairness by group and robustness under perturbation
- Check explainability and define drift-monitoring
- Gate on the thresholds; route failing results to triage, flag human/ethical review
Why it works
The theory behind it
CT-AI, mapping ISO/IEC 25010 (functional suitability, reliability, performance, security) extended with AI-specific characteristics (correctness under non-determinism, adaptability, autonomy, drift resistance, transparency, freedom from unwanted bias). Non-deterministic systems need metric-based, not exact-match, oracles.
CT-AI (Specialist) · ISO/IEC 25010 + AI-specific quality characteristics
Example
See it in use
> /qa:ai-test "product recommendations"✓ Correct when It covers data quality, model metrics, bias/fairness, robustness, and drift (CT-AI).