Kafka Prerequisites

EvidentSource requires an existing Apache Kafka cluster for distributed transaction coordination and event streaming. This page documents the prerequisites for Kafka integration.

Bring Your Own Kafka (BYOK)

EvidentSource does not provision a Kafka cluster as part of CloudFormation deployment. You must provide:

1. An existing Kafka cluster (Apache Kafka, AWS MSK, Confluent Cloud, etc.)
2. Network connectivity from your VPC to the Kafka brokers
3. Pre-created topics with appropriate configurations
4. Authentication credentials (if using secured Kafka)

Required Topics

EvidentSource uses four Kafka topics with distinct purposes and access patterns:

Topic	Default Name	Access	Purpose
Transaction Proposals	evidentsource-transaction-proposals	Internal	Distributed transaction coordination between cluster nodes
Database Events	evidentsource-database-events	Public Read	Notifications when databases are created, events transacted, or databases deleted
Async Commands	evidentsource-async-commands	Public Write	End-user submission of asynchronous commands
Async Responses	evidentsource-async-responses	Public Read	Delivery of asynchronous command results

Topic Access Patterns

Internal Topics should only be accessed by EvidentSource components:

• Transaction Proposals: Used by the Kafka linearizer for distributed consensus

Public Read Topics can be consumed by external applications:

• Database Events: Subscribe to receive notifications of all database changes
• Async Responses: Receive results of asynchronously submitted commands

Public Write Topics accept messages from external producers:

• Async Commands: Submit commands for asynchronous processing

Topic Configuration

Recommended Settings

Topic	Partitions	Replication Factor	Retention
Transaction Proposals	3+	3 (production)	1 day
Database Events	3+	3 (production)	7 days
Async Commands	3+	3 (production)	1 day
Async Responses	3+	3 (production)	1 day

Partitioning Strategy

• Transaction Proposals: Messages are keyed by database name for ordering guarantees
• Database Events: Messages are keyed by database name for per-database ordering
• Async Commands/Responses: Messages are keyed by correlation ID

Create Topics with Apache Kafka CLI

# Set your Kafka broker address
KAFKA_BROKERS="localhost:9092"

# Create all required topics
kafka-topics.sh --create \
  --topic evidentsource-transaction-proposals \
  --bootstrap-server $KAFKA_BROKERS \
  --partitions 3 \
  --replication-factor 3 \
  --if-not-exists

kafka-topics.sh --create \
  --topic evidentsource-database-events \
  --bootstrap-server $KAFKA_BROKERS \
  --partitions 3 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --if-not-exists

kafka-topics.sh --create \
  --topic evidentsource-async-commands \
  --bootstrap-server $KAFKA_BROKERS \
  --partitions 3 \
  --replication-factor 3 \
  --if-not-exists

kafka-topics.sh --create \
  --topic evidentsource-async-responses \
  --bootstrap-server $KAFKA_BROKERS \
  --partitions 3 \
  --replication-factor 3 \
  --if-not-exists

Create Topics with AWS MSK

# Set your MSK cluster ARN
MSK_CLUSTER_ARN="arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/..."

# Get bootstrap brokers
KAFKA_BROKERS=$(aws kafka get-bootstrap-brokers \
  --cluster-arn $MSK_CLUSTER_ARN \
  --query 'BootstrapBrokerStringSaslIam' \
  --output text)

# Create topics using kafka-topics.sh with IAM authentication
# (Requires MSK IAM auth configured in client.properties)
for topic in evidentsource-transaction-proposals \
             evidentsource-database-events \
             evidentsource-async-commands \
             evidentsource-async-responses; do
  kafka-topics.sh --create \
    --topic $topic \
    --bootstrap-server $KAFKA_BROKERS \
    --command-config client.properties \
    --partitions 3 \
    --replication-factor 3 \
    --if-not-exists
done

Access Control

IAM Permissions for AWS MSK

When using AWS MSK with IAM authentication, configure the following permissions:

EvidentSource Server Role (EC2 instances):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster"
      ],
      "Resource": "arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:ReadData",
        "kafka-cluster:WriteData",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:CreateTopic"
      ],
      "Resource": [
        "arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*/evidentsource-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:AlterGroup",
        "kafka-cluster:DescribeGroup"
      ],
      "Resource": "arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/*"
    }
  ]
}

DynamoDB Streams Lambda Role:

{
  "Effect": "Allow",
  "Action": [
    "kafka-cluster:Connect",
    "kafka-cluster:WriteData",
    "kafka-cluster:DescribeTopic"
  ],
  "Resource": [
    "arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/*",
    "arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*/evidentsource-database-events"
  ]
}

External Consumers (Database Events, Async Responses):

{
  "Effect": "Allow",
  "Action": [
    "kafka-cluster:Connect",
    "kafka-cluster:ReadData",
    "kafka-cluster:DescribeTopic",
    "kafka-cluster:AlterGroup",
    "kafka-cluster:DescribeGroup"
  ],
  "Resource": [
    "arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/*",
    "arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*/evidentsource-database-events",
    "arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*/evidentsource-async-responses",
    "arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/*"
  ]
}

External Producers (Async Commands):

{
  "Effect": "Allow",
  "Action": [
    "kafka-cluster:Connect",
    "kafka-cluster:WriteData",
    "kafka-cluster:DescribeTopic"
  ],
  "Resource": [
    "arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/*",
    "arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/*/evidentsource-async-commands"
  ]
}

Kafka ACLs (Non-IAM)

For Kafka clusters using traditional ACLs:

# EvidentSource server - full access to all topics
kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:evidentsource \
  --operation All \
  --topic evidentsource-transaction-proposals

kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:evidentsource \
  --operation All \
  --topic evidentsource-database-events

kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:evidentsource \
  --operation All \
  --topic evidentsource-async-commands

kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:evidentsource \
  --operation All \
  --topic evidentsource-async-responses

# External consumers - read from public topics
kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:external-consumer \
  --operation Read --operation Describe \
  --topic evidentsource-database-events

kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:external-consumer \
  --operation Read --operation Describe \
  --topic evidentsource-async-responses

# External producers - write to async commands
kafka-acls.sh --bootstrap-server $KAFKA_BROKERS \
  --add --allow-principal User:external-producer \
  --operation Write --operation Describe \
  --topic evidentsource-async-commands

Network Configuration

VPC Connectivity

Ensure your EvidentSource VPC can reach the Kafka brokers:

1. Same VPC: If Kafka is in the same VPC, ensure security groups allow traffic on Kafka ports (typically 9092, 9094, or 9096)

2. Different VPC: Use VPC peering or AWS PrivateLink to connect VPCs

3. AWS MSK:

   • Deploy MSK in the same VPC as EvidentSource, or
   • Use MSK multi-VPC connectivity (PrivateLink)
   • Ensure MSK security group allows inbound from EvidentSource security groups

Security Group Rules

MSK Security Group should allow inbound from:

• EvidentSource cluster security group
• EvidentSource Lambda security group (KafkaAccessSecurityGroup)

# Allow EvidentSource to access MSK
aws ec2 authorize-security-group-ingress \
  --group-id sg-msk-cluster \
  --source-group sg-evidentsource-kafka-access \
  --protocol tcp \
  --port 9096  # IAM auth port

Security Configuration

CloudFormation Parameters

The enterprise CloudFormation template accepts these security parameters:

Parameter	Description
KafkaSecurityProtocol	Security protocol: empty (PLAINTEXT), SSL, or SASL_SSL
KafkaSaslMechanism	SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, or AWS_MSK_IAM
KafkaSaslJaasConfig	JAAS configuration string for SCRAM/PLAIN auth (not needed for AWS_MSK_IAM)

AWS MSK with IAM Authentication

For AWS MSK using IAM authentication (recommended):

Parameters:
  KafkaBrokers: "b-1.mycluster.abc123.kafka.us-east-1.amazonaws.com:9098,b-2.mycluster.abc123.kafka.us-east-1.amazonaws.com:9098"
  KafkaSecurityProtocol: "SASL_SSL"
  KafkaSaslMechanism: "AWS_MSK_IAM"
  # KafkaSaslJaasConfig not needed - uses IAM role automatically

IAM authentication is handled automatically using the EC2 instance role or Lambda execution role. No JAAS configuration is required.

SCRAM Authentication

For Kafka using SCRAM authentication:

Parameters:
  KafkaBrokers: "broker1:9094,broker2:9094"
  KafkaSecurityProtocol: "SASL_SSL"
  KafkaSaslMechanism: "SCRAM-SHA-512"
  KafkaSaslJaasConfig: 'org.apache.kafka.common.security.scram.ScramLoginModule required username="evidentsource" password="your-secret-password";'

PLAIN Authentication

For Kafka using PLAIN authentication:

Parameters:
  KafkaBrokers: "broker1:9094,broker2:9094"
  KafkaSecurityProtocol: "SASL_SSL"
  KafkaSaslMechanism: "PLAIN"
  KafkaSaslJaasConfig: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="evidentsource" password="your-secret-password";'

Development (No Authentication)

For development environments without Kafka authentication:

Parameters:
  KafkaBrokers: "localhost:9092"
  # Leave security parameters empty for PLAINTEXT

Verification

After creating topics and configuring access, verify connectivity:

# List topics to verify creation
kafka-topics.sh --list \
  --bootstrap-server $KAFKA_BROKERS \
  --command-config client.properties

# Describe a topic to verify configuration
kafka-topics.sh --describe \
  --topic evidentsource-database-events \
  --bootstrap-server $KAFKA_BROKERS \
  --command-config client.properties

Next Steps

• Review CloudFormation Parameters for full deployment configuration
• Learn about Kafka Streaming API for consuming events
• See AWS Deployment for complete deployment guide