Tuning: throughput, latency, and S3 cost

EvidentSource stores everything in object storage through SlateDB, so tuning is a three-way trade between write latency, sustained throughput, and S3 request cost. This page explains the model first, because the knobs only make sense against it, then gives configurations that work.

All knobs are environment variables on the container (each also exists as a CLI flag).

The throughput model

Sustained write throughput is the minimum of three terms:

throughput = min( admission, loop ceiling, drain rate )

• Admission — how many transactions can complete per second given their latency: concurrency ÷ write latency. Write latency is dominated by the WAL flush interval (below), so at the default 30ms a client holding 400 concurrent writes can complete at most ~12,400/s no matter what else you tune.
• Loop ceiling — the write loop is a single sequential coalescer per database; its per-transaction CPU cost bounds throughput regardless of concurrency.
• Drain rate — how fast memtables can be flushed to L0 SSTs and compacted. When ingest outruns the drain, in-memory data accumulates until SlateDB pauses writers. A generous memory budget only decides how long the burst lasts — it never fixes a drain that’s genuinely behind.

Each section below tunes one term.

Write latency and durability: the WAL flush interval

EVIDENTSOURCE_WAL_FLUSH_INTERVAL_MS   (default: 30)

SlateDB makes writes durable by flushing its write-ahead log to object storage on a timer, so this interval is your durability window and the floor of your write latency:

write p50 ≈ flush_interval + one PUT round-trip

There is a second trigger: the WAL buffer also flushes early if it exceeds the L0 SST size threshold (EVIDENTSOURCE_L0_SST_SIZE_MB, 64MB default — the same knob that sizes L0 SSTs below). At typical event sizes the timer fires long before the size threshold (~1KB/transaction × a 30ms window is a few hundred KB), so the interval governs; the size trigger matters when large payloads meet a long interval, and it caps the PUT-cost estimate below.

The trade is PUT frequency: a loaded database performs one WAL PUT per interval (~33/s at 30ms) until ingest rate × interval approaches the size threshold. An idle database performs zero — the timer freezes nothing when nothing is buffered — so a low interval costs nothing at idle.

Raise it (100–200ms) when latency doesn’t matter and you want fewer PUTs under load; lower it only if you need tighter durability and are prepared to pay for the PUT rate.

The drain chain

Under sustained load, committed data flows: memtable → frozen → uploaded as an L0 SST → compacted into sorted runs. Four knobs govern it.

EVIDENTSOURCE_L0_MAX_SSTS_PER_KEY (default: 8)

This is the cap that actually binds — not EVIDENTSOURCE_L0_MAX_SSTS. SlateDB halts memtable flushes when any single key would be covered by more than this many L0 SSTs. EvidentSource memtables mix event-log rows and index rows, so every L0 SST spans essentially the whole keyspace and overlaps every other one: the per-key cap is reached at 8 total L0 SSTs, long before a total-count cap of 64 means anything.

When the cap is hit, flushes halt, unflushed data accumulates, and writers eventually pause. Raising it to 32 gives compaction room to catch up during bursts, at the cost of point-read amplification on keys that miss the memtable and WAL.

EVIDENTSOURCE_COMPACTOR_POLL_INTERVAL_MS (default: 30000)

How often the compactor checks for work. The default is deliberately high because each poll is one manifest GET per writer-database — a real cost floor for a fleet of mostly-idle databases (~$0.03/month per database at 30s, ~$0.21 at 5s).

But deferring compaction does not just shift work. Under sustained load, the per-key L0 cap fills, flushes halt — and a compactor that’s up to 30 seconds late turns that into periodic throughput collapses with multi-second writer pauses. Measured on a saturated database: 30s polling stalled a 12.3k tx/s workload to 2–5k on a 30-second cadence, with individual writes paused up to 18s; 5s polling removed the stalls entirely.

Rule of thumb: 5000 for any database that sees sustained writes; 30000 is a cost optimization for idle fleets only. A write-hot database doesn’t notice the poll cost — it’s already doing an order of magnitude more PUTs than polls.

EVIDENTSOURCE_L0_FLUSH_PARALLELISM (default: 4)

Parallel workers uploading frozen memtables to L0. Raising to 8 overlaps object-store upload latency; it changes concurrency, not total PUT count.

EVIDENTSOURCE_L0_SST_SIZE_MB (default: 64)

The memtable size at which it is frozen and uploaded as one L0 SST. Larger (128) means fewer, bigger uploads and less pressure on the per-key cap. The costs are longer WAL replay on recovery and higher visibility lag for non-writer readers, which only see data once it reaches L0.

EVIDENTSOURCE_MEMTABLE_BUDGET_MB (default: 256)

Total unflushed (WAL + memtable) bytes allowed in memory before SlateDB pauses writers. This is a shock absorber, not a throughput knob: it should be large enough to ride out one compaction cycle, but if your drain rate is below your ingest rate, a bigger budget only delays the same collapse.

Dual-bucket: fast WAL, cheap data

EVIDENTSOURCE_S3_BUCKET       # compacted data: general-purpose S3
EVIDENTSOURCE_WAL_BUCKET      # WAL: S3 Express One Zone

Setting EVIDENTSOURCE_WAL_BUCKET is the whole switch: the WAL bucket automatically uses S3 Express addressing and the compacted bucket uses standard addressing. (EVIDENTSOURCE_S3_EXPRESS applies only to single-bucket deployments, where it switches the one bucket to Express addressing — don’t set it here.)

The WAL is the latency-critical path (every write waits on a WAL PUT) and its objects are small and short-lived; compacted data is the throughput path and lives forever. Splitting them puts each on the right storage class: S3 Express One Zone gives single-digit-millisecond WAL PUTs, while L0 uploads and compaction run against a standard bucket where storage is cheap.

Idle-fleet S3 cost floors

Every writer database polls object storage even when idle. These four intervals set the baseline request rate per database — multiply by your database count before lowering any of them:

knob	default	what each tick costs
EVIDENTSOURCE_READER_REFRESH_INTERVAL_MS	1000	1 manifest GET per reader node
EVIDENTSOURCE_WRITER_MANIFEST_POLL_INTERVAL_MS	10000	1 manifest GET (fence detection)
EVIDENTSOURCE_COMPACTOR_POLL_INTERVAL_MS	30000	1 manifest GET
EVIDENTSOURCE_GC_INTERVAL_MS	300000	4 LISTs (one per GC directory)

Two knobs that are not what they look like

EVIDENTSOURCE_MAX_BATCH_TRANSACTIONS (default: 8) — the write loop’s coalescer cap: how many queued transactions commit in one batch. It does not control SST sizes or fragmentation (the flush interval and L0 SST size do), and raising it doesn’t raise throughput in flush-bound regimes. It’s a per-commit CPU-overhead knob; 64 is safe and measured throughput-neutral.

Client-supplied transaction ids — not an env var, but a real throughput lever. A transaction that arrives without a transaction_id gets a server-minted one that is provably fresh, so the write loop skips its idempotency probe. A client-supplied id must be probed against the transaction index — a negative point-get whose cost grows with store size and degrades sustained throughput as the database grows. Use the plain transact SDK calls (which omit the id) unless you specifically need client-controlled idempotency via *_with_id.

DataFusion query settings

The server and the read peer build their DataFusion sessions from DataFusion’s own environment variables: DATAFUSION_<SETTING> for any session setting datafusion.<setting>, upper-cased with dots as underscores (for example DATAFUSION_EXECUTION_BATCH_SIZE). They override the server’s own defaults, which are tuned for low-latency point queries (batch_size 1024, target_partitions 1). DataFusion’s runtime settings (DATAFUSION_RUNTIME_*) are not read: EvidentSource builds the runtime environment itself. A DATAFUSION_* variable that matches no setting read here, such as a misspelling or a runtime setting, is logged as a warning at startup and otherwise ignored. That warning goes through tracing, so it appears wherever the process sends its logs — the server and evidentsource-sql both install a subscriber. An embedding host that installs none shows nothing.

Some settings ignore the environment:

• datafusion.optimizer.repartition_file_scans stays false everywhere. Repartitioning a cold-segment scan would split each segment by its recorded size, so a wrong recorded size could silently drop rows.
• The read peer’s SQL sessions set their own information schema (on), their default catalog and schema, the SQL parser’s dialect and recursion limit (DataFusion’s defaults), and, when the session is given one, a target-partitions limit. evidentsource-sql sets no such limit, so there DATAFUSION_EXECUTION_TARGET_PARTITIONS applies.

If DataFusion can’t parse a value, or DATAFUSION_EXECUTION_BATCH_SIZE is 0, the process fails at startup with an error naming the variable.

The cold-segment scan also reads these Parquet settings:

Variable	Default	Effect on cold reads
DATAFUSION_EXECUTION_PARQUET_ENABLE_PAGE_INDEX	true	Page-level pruning. A selective query pays one page-index read per segment, and it isn’t cached. On small single-page segments it prunes nothing, so false saves that read.
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS	false	Filters rows while decoding. In the #251 benchmark (with ..._REORDER_FILTERS) it helped two of the six queries measured, both on small segments: a selective query (58% fewer bytes, 136 → 122 ms) and a projection (148 → 127 ms). It slowed the other four, by up to 2x (an unselective filtered scan of large segments, reading every column: 952 → 1,905 ms). Leave it off unless you’ve measured your workload.
DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS	false	Only matters with pushdown on.
DATAFUSION_EXECUTION_PARQUET_METADATA_SIZE_HINT	8192 (EvidentSource’s)	Bytes prefetched from the end of each segment to read its footer. You can’t choose DataFusion’s own default of 524288: that exact value counts as unset and becomes 8192.
DATAFUSION_EXECUTION_PARQUET_PRUNING, ..._BLOOM_FILTER_ON_READ, ..._MAX_PREDICATE_CACHE_SIZE	DataFusion’s	Same meaning as in DataFusion. Segments don’t carry bloom filters today.

Configurations that work

Throughput profile — sustained high write rates on one database. Measured sustaining 12,000+ tx/s flat (p50 ≈ 32ms) where the defaults stalled to an 8,400 tx/s mean with multi-second pauses:

EVIDENTSOURCE_WAL_FLUSH_INTERVAL_MS=30
EVIDENTSOURCE_COMPACTOR_POLL_INTERVAL_MS=5000
EVIDENTSOURCE_L0_MAX_SSTS_PER_KEY=32
EVIDENTSOURCE_L0_FLUSH_PARALLELISM=8
EVIDENTSOURCE_L0_SST_SIZE_MB=128
EVIDENTSOURCE_MEMTABLE_BUDGET_MB=1024
EVIDENTSOURCE_MAX_BATCH_TRANSACTIONS=64

Cost profile — many mostly-idle databases, latency-tolerant:

EVIDENTSOURCE_WAL_FLUSH_INTERVAL_MS=200
EVIDENTSOURCE_COMPACTOR_POLL_INTERVAL_MS=30000
EVIDENTSOURCE_READER_REFRESH_INTERVAL_MS=10000
EVIDENTSOURCE_WRITER_MANIFEST_POLL_INTERVAL_MS=10000
EVIDENTSOURCE_GC_INTERVAL_MS=300000

The profiles differ where the trade-offs are real: the throughput profile accepts ~$0.20/month/database of extra polling to eliminate stalls; the cost profile accepts 200ms write latency and slower compaction on the bet that no database sees sustained load. If one fleet contains both kinds of database, configure per deployment, not per fleet.