115 controls · 9 dimensions · Grounded in NIST, AWS, Microsoft & Google frameworks

9 Dimensions of AI Production Readiness

Every AI workflow ShipSmith assesses is scored against these 9 dimensions. Each dimension maps to a set of controls grounded in established industry frameworks.

Assess Your Workflows →
01

Data Foundation

The AI is only as good as what it's fed. Most production failures trace back here.

Google MLOps Maturity ModelAWS ML LensNIST AI RMF
14 controls

Common failures in production

No pipeline — data is exported manually before each run
PII in training data was never identified
Ground-truth labels don't exist — accuracy can't be measured
Data freshness is assumed, not monitored
02

Model & Architecture

Is the right model being used the right way, or did someone just pick the default?

Google MLOps Maturity ModelAWS GenAI LensMicrosoft Responsible AI
15 controls

Common failures in production

System prompts are hardcoded in application code, not version-controlled
Model was chosen because it was popular, not because it was benchmarked
No upgrade path documented when the model is deprecated
RAG retrieval strategy was never tuned — using defaults
03

Evaluation & QA

The single biggest dividing line between production systems and wrappers.

Microsoft Responsible AI (Fairlearn)AWS GenAI LensNIST AI RMF
15 controls

Common failures in production

No eval dataset — accuracy is assessed manually after incidents
Evals exist but don't block deployment
Hallucination rate has never been measured
No adversarial test cases
04

Observability & Monitoring

You cannot improve what you cannot see. Most teams are flying blind.

AWS GenAI LensAWS ML LensMicrosoft Responsible AI
14 controls

Common failures in production

LLM calls are not logged — no way to debug failures
Cost per request is unknown
No alerting on error rate spikes
No dashboard for business stakeholders
05

Resilience & Production Engineering

Does this system survive the real world, or only the demo environment?

Google MLOps Level 1-2AWS ML LensAWS GenAI Lens
13 controls

Common failures in production

No fallback — if the LLM is down, the feature is down
No retry logic — transient 429s cause user-facing errors
Load testing has never been done
No rollback procedure for bad prompt deployments
06

Security & Compliance

Often the last thing AI builders think about, and the first thing enterprise buyers ask about.

OWASP LLM Top 10AWS GenAI LensMicrosoft Responsible AI
14 controls

Common failures in production

API keys in environment files, not a secrets manager
No prompt injection mitigations
Agentic tool calls not sandboxed — can take irreversible actions
No incident response plan for AI-specific failures
07

Cost Management

Unchecked AI costs are one of the most common reasons companies kill promising AI projects.

AWS GenAI LensGoogle MLOps Level 1
11 controls

Common failures in production

No token budgets — a single runaway agent can cost hundreds
No alerting on spend spikes
Simple tasks use expensive models — routing hasn't been considered
No cost model for what happens at 10x usage
08

Adoption & Change Management

70% of AI success is people, process, and culture. This is the most neglected dimension.

NIST AI RMFMicrosoft Responsible AI
13 controls

Common failures in production

No named owner — if something breaks, nobody knows who to call
No feedback channel — users can't report AI errors
Adoption rate is unknown — nobody checks if people use it
Leadership has no visibility into how the system is performing
09

AI Governance & Risk Management

The organisational scaffolding that determines whether AI is run responsibly at scale.

NIST AI RMFAWS GenAI LensMicrosoft Responsible AI Standard v2
6 controls

Common failures in production

No written AI policy — acceptable use is implicit, not defined
No impact assessment before deployment
Third-party vendors' data retention policies are unknown
No named AI risk owner — the product owner carries all risk

See how your AI workflows score across all 9 dimensions.

Free for your first workflow. No credit card required.

Scan Your Repo — Free →