Storage Adapters

EvidentSource uses a pluggable storage architecture that allows different backends while maintaining the same API and semantics. This design enables you to choose the right storage solution for your use case, from in-memory for development to distributed cloud storage for production.

Storage Adapter Interface

All storage adapters implement a common interface:

#[async_trait]
pub trait StorageAdapter: Send + Sync {
    // Database lifecycle
    async fn create_database(&self, name: &str) -> Result<()>;
    async fn delete_database(&self, name: &str) -> Result<()>;
    async fn list_databases(&self) -> Result<Vec<DatabaseInfo>>;

    // Event operations
    async fn append_batch(&self, db: &str, batch: EventBatch) -> Result<Revision>;
    async fn read_events(&self, db: &str, range: RevisionRange) -> Result<Vec<Event>>;

    // Index operations
    async fn query_index(&self, db: &str, query: IndexQuery) -> Result<Vec<Revision>>;
    async fn update_indexes(&self, db: &str, events: &[Event]) -> Result<()>;

    // State management
    async fn get_latest_revision(&self, db: &str) -> Result<Revision>;
    async fn get_database_info(&self, db: &str) -> Result<DatabaseInfo>;
}

Available Adapters

In-Memory Adapter

Perfect for development and testing:

evidentsource-server --scheme http in-memory

Characteristics:

• All data stored in RAM
• Extremely fast performance
• No persistence between restarts
• Single-node only
• Great for unit tests

Use Cases:

• Local development
• Unit and integration testing
• Prototyping and demos
• CI/CD pipelines

Configuration:

InMemoryAdapter::new(InMemoryConfig {
    max_databases: 100,
    max_events_per_database: 1_000_000,
})

DynamoDB + SNS Adapter

Production-ready AWS deployment:

evidentsource-server --scheme http dynamodb \
  --table-prefix evident \
  --sns-topic-arn arn:aws:sns:us-east-1:123456789012:evident-events

Characteristics:

• Serverless, managed storage
• Automatic scaling
• High availability (multi-AZ)
• Pay-per-request pricing option
• Real-time event streaming via SNS

Architecture:

graph LR
    A[EvidentSource Server] --> B[DynamoDB Tables]
    A --> C[SNS Topic]
    B --> D[Events Table]
    B --> E[Indexes Table]
    B --> F[Databases Table]
    C --> G[Lambda Functions]
    C --> H[SQS Queues]
    C --> I[Event Consumers]

Tables Structure:

Events Table:

Attribute	Type	Description
database_name (PK)	String	Database identifier
revision (SK)	Number	Event revision number
event_data	Binary	Serialized CloudEvent
recorded_time	String	ISO timestamp
stream	String	Event source (GSI)
subject	String	Event subject (GSI)
event_type	String	Event type (GSI)

Indexes Table:

Attribute	Type	Description
database_name (PK)	String	Database identifier
index_key (SK)	String	Composite index key
revisions	List	Revision numbers
last_updated	String	Update timestamp

Configuration:

dynamodb:
  table_prefix: evident
  read_capacity: 5
  write_capacity: 5
  billing_mode: PAY_PER_REQUEST
  point_in_time_recovery: true

sns:
  topic_arn: arn:aws:sns:region:account:topic
  enable_raw_message_delivery: true

DynamoDB + Kafka Adapter

Alternative streaming with Kafka:

evidentsource-server --scheme http dynamodb-kafka \
  --kafka-brokers localhost:9092 \
  --kafka-topic evident-events

Characteristics:

• Same DynamoDB storage as above
• Kafka for event streaming
• Better for high-throughput scenarios
• Supports consumer groups and replay
• More complex operational requirements

When to Use:

• High event throughput (>10k events/sec)
• Need for event replay
• Multiple consumer groups
• Existing Kafka infrastructure

Configuration:

kafka:
  brokers:
    - broker1:9092
    - broker2:9092
  topic: evident-events
  replication_factor: 3
  compression_type: snappy
  acks: all

Storage Adapter Selection

Decision Matrix

Criteria	In-Memory	DynamoDB+SNS	DynamoDB+Kafka
Setup Complexity	None	Medium	High
Operational Overhead	None	Low	High
Scalability	Single Node	Auto-scaling	Manual scaling
Durability	None	High	High
Cost	Free	Pay-per-use	Fixed + usage
Latency	<1ms	5-10ms	10-20ms
Throughput	Very High	Medium	High
Event Streaming	No	Yes (SNS)	Yes (Kafka)

Choosing the Right Adapter

Use In-Memory when:

• Developing locally
• Running tests
• Data loss is acceptable
• Single-node is sufficient

Use DynamoDB+SNS when:

• Deploying to AWS
• Want serverless operations
• Need simple event streaming
• Moderate throughput is sufficient

Use DynamoDB+Kafka when:

• Very high throughput required
• Complex event processing
• Need event replay capability
• Have Kafka expertise

Custom Storage Adapters

You can implement custom storage adapters:

Implementation Guide

use evidentsource::storage::{StorageAdapter, Result};

pub struct MyCustomAdapter {
    // Your storage backend
}

#[async_trait]
impl StorageAdapter for MyCustomAdapter {
    async fn create_database(&self, name: &str) -> Result<()> {
        // Implement database creation
        todo!()
    }

    async fn append_batch(&self, db: &str, batch: EventBatch) -> Result<Revision> {
        // Implement atomic batch append
        // Must ensure:
        // 1. All events get consecutive revisions
        // 2. Either all succeed or all fail
        // 3. Return the last revision assigned
        todo!()
    }

    // ... implement other methods
}

Requirements

Custom adapters must ensure:

1. Atomicity: Batch operations are all-or-nothing
2. Consistency: Strong consistency for reads after writes
3. Isolation: Database operations are isolated
4. Durability: Committed events are not lost
5. Ordering: Revisions are strictly monotonic

Testing Your Adapter

Use the provided test suite:

#[cfg(test)]
mod tests {
    use evidentsource::storage::tests::*;

    #[test]
    fn test_my_adapter() {
        let adapter = MyCustomAdapter::new();
        run_storage_adapter_tests(adapter);
    }
}

Performance Considerations

In-Memory Performance

• Writes: ~1 million events/sec
• Reads: ~10 million events/sec
• Memory Usage: ~100 bytes per event
• Startup Time: Instant

DynamoDB Performance

• Writes: Up to 40k events/sec (with proper scaling)
• Reads: Up to 40k queries/sec
• Latency: p50: 5ms, p99: 20ms
• Auto-scaling: Based on consumed capacity

Optimization tips:

# Enable auto-scaling
auto_scaling:
  target_utilization: 70
  min_capacity: 5
  max_capacity: 40000

# Use on-demand for variable workloads
billing_mode: PAY_PER_REQUEST

# Enable point-in-time recovery
point_in_time_recovery: true

Kafka Performance

• Throughput: 100k+ events/sec
• Latency: 10-50ms depending on configuration
• Durability: Configurable via replication

Optimization tips:

# Producer settings
producer:
  batch_size: 16384
  linger_ms: 10
  compression_type: lz4

# Topic settings
topic:
  partitions: 50
  replication_factor: 3
  min_insync_replicas: 2

Migration Between Adapters

Export/Import

Export from one adapter and import to another:

# Export from in-memory
evidentsource-tool export \
  --source in-memory \
  --database mydb \
  --output mydb-export.json

# Import to DynamoDB
evidentsource-tool import \
  --target dynamodb \
  --database mydb \
  --input mydb-export.json

Live Migration

For zero-downtime migration:

1. Dual-write: Write to both adapters
2. Backfill: Copy historical data
3. Verify: Ensure consistency
4. Cutover: Switch reads to new adapter
5. Cleanup: Remove old adapter

Monitoring Storage

Metrics to Track

All Adapters:

• Event append latency
• Query latency
• Error rates
• Storage size

DynamoDB Specific:

• Consumed read/write units
• Throttled requests
• System errors
• Item count

Kafka Specific:

• Producer/consumer lag
• Partition distribution
• Replication status
• Disk usage

Health Checks

// Built-in health check endpoint
GET /health/storage

// Response
{
  "adapter": "dynamodb",
  "status": "healthy",
  "databases": 5,
  "total_events": 1234567,
  "latency_ms": {
    "write_p50": 5,
    "write_p99": 20,
    "read_p50": 3,
    "read_p99": 15
  }
}

Best Practices

General Guidelines

1. Choose based on requirements: Don’t over-engineer
2. Monitor actively: Set up alerts for anomalies
3. Plan for growth: Consider future scaling needs
4. Test failover: Ensure recovery procedures work
5. Backup regularly: Even with durable storage

DynamoDB Best Practices

• Use on-demand billing for unpredictable workloads
• Enable point-in-time recovery
• Set up cross-region replication for DR
• Use VPC endpoints for security
• Monitor consumed capacity closely

Kafka Best Practices

• Size partitions based on throughput
• Use appropriate replication factor (usually 3)
• Configure retention based on replay needs
• Monitor consumer lag closely
• Use compression to reduce costs

Future Adapters

Planned storage adapters:

• PostgreSQL: For SQL-based deployments
• S3 + Parquet: For analytical workloads
• Redis: For ultra-low latency caching
• FoundationDB: For global distribution

Next Steps

• Learn about the WebAssembly Runtime
• Explore AWS Deployment for production
• See Local Development setup
• Check Performance Tuning guide