Simulation Fidelity Matrix

Understanding simulation accuracy across different services and tiers.

Overview

Shadow Executor simulates AWS operations before execution to predict outcomes without touching real infrastructure. Simulation fidelity varies by service, operation, and simulation tier.

Simulation Tiers

Tier	Approach	Fidelity	Latency	Cost	Availability
Local (Milestone 1)	LocalStack + mocks	60-80%	< 100ms	Free	Available
Cloud (Milestone 2)	Ephemeral AWS stacks	95-99%	5-30s	$0.01-$0.50/simulation	Q3 2026
Production (Milestone 3)	Real AWS with rollback	100%	Variable	Variable	Q4 2026

Local Simulation (Current)

Uses LocalStack and custom mocks.

S3 Operations

Operation	Fidelity	Notes
CreateBucket	90%	Bucket naming, ACLs supported. Cross-region replication not simulated.
PutObject	85%	Object storage, metadata, tagging supported. Glacier transitions not simulated.
GetObject	90%	Content retrieval, versioning supported.
DeleteObject	95%	Deletion, versioning, lifecycle rules supported.
DeleteBucket	90%	Bucket deletion with object cleanup. MFA delete not simulated.
ListObjects	85%	Pagination, filters supported.
PutBucketAcl	80%	Public access settings. Bucket policies partially supported.

Limitations:

• No Glacier storage class simulation
• No S3 Select simulation
• Bucket policies: basic support only

RDS Operations

Operation	Fidelity	Notes
CreateDBInstance	75%	Instance creation, parameter groups. Multi-AZ not simulated.
DeleteDBInstance	90%	Instance deletion, snapshot creation.
ModifyDBInstance	70%	Parameter changes. Performance impact not simulated.
DescribeDBInstances	85%	Instance metadata, status.
CreateDBSnapshot	80%	Snapshot creation. Automated backups partially supported.

Limitations:

• No actual database engine running (MySQL, Postgres, etc.)
• No read replica simulation
• No performance metrics
• Multi-AZ deployments return success but don't create actual standby

DynamoDB Operations

Operation	Fidelity	Notes
CreateTable	85%	Table creation, key schema, provisioned throughput.
PutItem	90%	Item writes, conditional puts, TTL supported.
GetItem	95%	Item reads, consistent reads supported.
UpdateItem	85%	Item updates, atomic counters, conditional updates.
DeleteItem	90%	Item deletion, conditional deletes.
Scan	80%	Full table scans, filters, pagination.
Query	85%	Index queries, sort key conditions.

Limitations:

• No DynamoDB Streams simulation
• No global tables simulation
• Capacity units not enforced
• No point-in-time recovery

Lambda Operations

Operation	Fidelity	Notes
CreateFunction	70%	Function creation, runtime selection. Layers not fully supported.
InvokeFunction	75%	Synchronous invocations in isolated Node.js VM.
DeleteFunction	90%	Function deletion.
UpdateFunctionCode	70%	Code updates. Versioning partially supported.

Limitations:

• Only Node.js runtime supported in Milestone 1
• No VPC integration simulation
• No Lambda@Edge
• Execution time limits not enforced
• Memory limits not enforced
• No CloudWatch Logs integration

IAM Operations

Operation	Fidelity	Notes
AttachUserPolicy	60%	Policy attachments tracked but not enforced.
CreateRole	70%	Role creation, assume role policies.
PutUserPolicy	60%	Inline policies tracked but not enforced.

Limitations:

• Policies not actually enforced in LocalStack
• No cross-account access simulation
• Trust relationships not fully simulated

Cloud Simulation Tier (Milestone 2)

Status: Coming Q3 2026

Uses ephemeral AWS stacks (SAM/CDK) with automatic tear-down.

Improvements Over Local

• Real AWS SDKs: Actual AWS API calls in isolated account
• 99% fidelity: Nearly identical to production
• All services: S3, RDS, DynamoDB, Lambda, IAM, EC2, ECS, etc.
• Automatic cleanup: Stacks torn down after simulation

Cost Estimate

• S3 operations: $0.005-$0.01/simulation
• RDS operations: $0.10-$0.50/simulation (micro instance spin-up)
• Lambda operations: $0.0001/simulation
• DynamoDB operations: $0.001-$0.01/simulation

Monthly estimate for 1000 simulations/month: $50-$200

Latency

• Simple operations (S3 PutObject): 5-10 seconds
• Complex operations (RDS CreateDBInstance): 10-30 seconds
• Multi-service workflows: 30-60 seconds

Choosing a Simulation Tier

Use Case	Recommended Tier
Development, testing policies	Local
CI/CD integration, fast feedback	Local
Pre-production validation	Cloud (Milestone 2)
Production deployment preview	Cloud (Milestone 2)
High-stakes operations (prod DB migration)	Production (Milestone 3)

Fidelity Reports

Shadow Executor includes fidelity in each simulation result:

{
  "action": "BLOCK",
  "simulation": {
    "tier": "local",
    "fidelity": 0.85,
    "predicted_outcome": "DatabaseDeleted",
    "simulated_at": "2026-05-06T12:00:00.000Z",
    "duration_ms": 42
  }
}

Improving Fidelity

Local Tier Improvements (Roadmap)

• Support for Aurora serverless
• Improved IAM policy evaluation
• DynamoDB Streams simulation
• Python Lambda runtime support
• VPC simulation for Lambda

Cloud Tier Development (Milestone 2)

• CDK stack generator for AWS services
• Automatic resource tagging for cleanup
• Cost tracking per simulation
• Parallel simulation support
• Simulation result caching

Testing Simulation Accuracy

Compare simulation predictions with actual outcomes:

import { simulateAction, executeAction } from '@shadow-executor/core';

// Run simulation
const simResult = await simulateAction({
  service: 's3',
  operation: 'PutObject',
  parameters: { Bucket: 'test-bucket', Key: 'test.txt' },
});

// Execute actual operation
const actualResult = await executeAction({
  service: 's3',
  operation: 'PutObject',
  parameters: { Bucket: 'test-bucket', Key: 'test.txt' },
});

// Compare
const match = compareResults(simResult, actualResult);
console.log(`Fidelity: ${match.score * 100}%`);

Next Steps

• Quickstart Guide — Get started with local simulation
• Policy Reference — Define policies based on simulation results
• API Reference — Programmatic simulation API