Bi-Temporal Indexing

EvidentSource provides bi-temporal indexing, allowing you to query your data across two time dimensions: when events occurred in the real world (effective time) and when they were recorded in the system (revision time). This powerful feature enables complex temporal queries essential for audit trails, compliance, and historical analysis.

The Two Time Dimensions

Effective Time (Business Time)

Effective time represents when an event actually occurred in the real world:

• Stored in the CloudEvents time attribute
• Represents the business reality
• Can be in the past, present, or even future
• May be corrected retroactively

Revision Time (System Time)

Revision time represents when an event was recorded in EvidentSource:

• Automatically assigned by the system
• Stored as recordedtime and revision number
• Always moves forward (append-only)
• Cannot be changed once recorded

Why Bi-Temporal?

Bi-temporal data modeling is crucial for:

Compliance and Auditing

• Regulatory requirements: Many industries require both when something happened and when it was recorded
• Audit trails: Complete history of data changes and when they were made
• Data lineage: Track how information evolved over time

Retroactive Corrections

• Late-arriving data: Record events that happened in the past
• Corrections: Fix errors while maintaining the original records
• Backdating: Handle scenarios where business events are recorded after they occur

Historical Analysis

• Point-in-time queries: “What did we know at this point?”
• As-of queries: “What was the state as of this business date?”
• Temporal joins: Correlate data across different time dimensions

Temporal Query Patterns

Query by Revision (System Time)

Query the database as it existed at a specific revision:

// Get database state at revision 1000
val database = client.fetchDatabaseAtRevision(
    databaseName = "mydb",
    revision = 1000
)

// Query events up to revision 1000
val query = EventQuery(
    revisionRange = RevisionRange(
        start = 1,
        end = 1000
    )
)

Query by Effective Time (Business Time)

Query events based on when they occurred in the real world:

// Get all events that occurred on a specific date
val query = EventQuery(
    effectiveTimeRange = TimeRange(
        start = "2024-01-01T00:00:00Z",
        end = "2024-01-02T00:00:00Z"
    )
)

// Database state as of a business date
val database = client.databaseEffectiveAtTimestamp(
    databaseName = "mydb",
    timestamp = "2024-01-15T12:00:00Z"
)

Combined Temporal Queries

Query across both time dimensions:

// "What did we know on Jan 31st about events that occurred in January?"
val query = EventQuery(
    // Business time: Events that occurred in January
    effectiveTimeRange = TimeRange(
        start = "2024-01-01T00:00:00Z",
        end = "2024-02-01T00:00:00Z"
    ),
    // System time: As recorded by Jan 31st
    revisionRange = RevisionRange(
        start = 1,
        end = revisionAsOfJan31
    )
)

Real-World Examples

Financial Transactions

Consider a bank transaction that occurred on Friday but was processed on Monday:

{
  "id": "txn-123",
  "type": "AccountDebit",
  "time": "2024-01-12T15:30:00Z",  // Friday (effective time)
  "data": {
    "amount": 100.00,
    "account": "ACC-789"
  }
}
// Recorded on Monday with revision 5678
// recordedtime: "2024-01-15T09:00:00Z"

Queries:

• “Show Friday’s transactions” → Query by effective time
• “What transactions were in the system by Monday morning?” → Query by revision
• “What Friday transactions did we know about by end of day Friday?” → Combined query

Inventory Adjustments

A warehouse discovers a counting error from last month:

{
  "id": "adj-456",
  "type": "InventoryAdjustment",
  "time": "2024-01-05T10:00:00Z",  // When the error occurred
  "subject": "SKU-123",
  "data": {
    "quantity": -50,
    "reason": "Counting error discovered"
  }
}
// Recorded today with revision 9012
// recordedtime: "2024-02-15T14:30:00Z"

This allows:

• Accurate historical inventory levels (effective time)
• Audit trail of when corrections were made (revision time)
• Analysis of how long it took to discover errors

Regulatory Reporting

Generate reports showing what was known at specific points:

// Quarterly report: What we knew at quarter end
val q4EndRevision = getRevisionAsOf("2023-12-31T23:59:59Z")

// Get all events for Q4 as known at quarter end
val q4Report = client.queryEvents(
    EventQuery(
        effectiveTimeRange = TimeRange(
            start = "2023-10-01T00:00:00Z",
            end = "2024-01-01T00:00:00Z"
        ),
        revisionRange = RevisionRange(
            start = 1,
            end = q4EndRevision
        )
    )
)

Temporal Consistency

Revision Consistency

• Revisions are strictly increasing
• Each batch gets a unique revision
• Queries at a revision see a consistent snapshot

Effective Time Considerations

• Events can have any effective time
• No ordering enforced on effective time
• Late events don’t affect revision order

State View Temporality

State Views can leverage both dimensions:

// State view that considers both times
fn reduce(state: State, event: Event) -> State {
    match event.time {
        Some(effective_time) => {
            // Business logic based on when event occurred
        }
        None => {
            // Use recorded time as fallback
        }
    }
}

Best Practices

Choose the Right Time

• User actions: Use effective time (when user clicked)
• System events: Consider using recorded time
• External events: Use effective time from source system
• Corrections: Always use original effective time

Handle Missing Times

Not all events have effective time:

val effectiveTime = event.time ?: event.extensions["recordedtime"]

Time Zone Considerations

• Always use UTC for storage
• Convert to local time zones for display
• Include timezone context in event data if needed

Temporal Indexes

EvidentSource automatically maintains indexes for efficient temporal queries:

• Revision index: O(1) lookup by revision
• Time index: Efficient range queries by effective time
• Combined indexes: Optimized for bi-temporal queries

Advanced Patterns

Temporal Joins

Join events across time dimensions:

// Find all orders and their payments as of month end
val monthEndRevision = getRevisionAsOf("2024-01-31T23:59:59Z")

val orders = queryEvents(
    EventSelector(eventType = "OrderCreated"),
    revisionRange = RevisionRange(end = monthEndRevision)
)

val payments = queryEvents(
    EventSelector(eventType = "PaymentReceived"),
    revisionRange = RevisionRange(end = monthEndRevision)
)

// Join by subject (order ID)

Temporal Aggregations

Aggregate across time windows:

// Daily totals as known at day end
fun dailyTotals(date: LocalDate): Map<String, Double> {
    val dayEnd = date.atTime(23, 59, 59)
    val revision = getRevisionAsOf(dayEnd)

    return queryEvents(
        effectiveTimeRange = TimeRange(
            start = date.atStartOfDay(),
            end = dayEnd
        ),
        revisionRange = RevisionRange(end = revision)
    ).groupBy { it.type }
     .mapValues { calculateTotal(it.value) }
}

Temporal Snapshots

Create point-in-time snapshots:

// Snapshot of all active orders as of a specific time
data class Snapshot(
    val effectiveTime: Instant,
    val revision: Long,
    val activeOrders: List<Order>
)

fun createSnapshot(effectiveTime: Instant): Snapshot {
    val revision = getCurrentRevision()
    val activeOrders = buildActiveOrders(effectiveTime, revision)

    return Snapshot(effectiveTime, revision, activeOrders)
}

Next Steps

• Learn about State Views for temporal materialization
• Explore Event Sourcing patterns
• See API Reference for temporal query APIs
• Understand Architecture of temporal indexes