Package com.linkedin.davinci.kafka.consumer

Class PartialUpdateAmplificationDetector

java.lang.Object

com.linkedin.davinci.kafka.consumer.PartialUpdateAmplificationDetector

public class PartialUpdateAmplificationDetector
extends Object

Per-partition detector for partial-update amplification. Tracks partition-level aggregates and a bounded set of the heaviest keys (by total result bytes) within a configurable reporting window.

Two-Level Design

• Level 1 (always on): O(1) per-event partition aggregates — total partial-update count, total result bytes, and count of large results exceeding the threshold.
• Level 2 (large results only): Bounded HashMap of up to MAX_TRACKED_KEYS keys, tracking per-key count, total request/result bytes, and max result size. Only activated when a partial-update result exceeds the configured threshold.

Reporting

Reporting is piggy-backed on the partial-update path: each recordAndMaybeReport(byte[], int, int, int) call atomically records the event and, if the window has elapsed, builds a snapshot and resets the window. This produces at most one summary log per partition per window — not per key, not per event.

Threading

All public methods are synchronized. The critical section runs on every partial-update event (O(1) counter increments and a System.currentTimeMillis() call), but the heavier Level 2 map operations only execute for large-result events (a small fraction of total partial updates). Partial-update processing may be parallel when isAAWCWorkloadParallelProcessingEnabled is true; contention is negligible given the ~10ns critical path for non-large results.

Memory

~1.5 KB per partition with active large results (Level 1: 40 bytes, Level 2: 20 entries × ~70 bytes). Level 2 is lazily allocated only when the first large result is detected.

Constructor Summary

Constructors

Constructor	Description
PartialUpdateAmplificationDetector(long reportIntervalMs)

Method Summary

Modifier and Type	Method	Description
com.linkedin.davinci.kafka.consumer.PartialUpdateAmplificationDetector.AmplificationReport	recordAndMaybeReport(byte[] keyBytes, int requestSizeBytes, int resultSizeBytes, int largeResultThreshold)	Record a partial-update event and, if the reporting window has elapsed, atomically build a report and reset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

PartialUpdateAmplificationDetector

public PartialUpdateAmplificationDetector(long reportIntervalMs)

Method Details

recordAndMaybeReport

public com.linkedin.davinci.kafka.consumer.PartialUpdateAmplificationDetector.AmplificationReport recordAndMaybeReport(byte[] keyBytes,
 int requestSizeBytes,
 int resultSizeBytes,
 int largeResultThreshold)

Record a partial-update event and, if the reporting window has elapsed, atomically build a report and reset. If the result size exceeds the threshold, the key is tracked in the heavy key map.

Combining record + report in a single synchronized method eliminates the race window between separate record and report calls in the parallel AA partial-update path.

Parameters:

keyBytes - the key bytes of the record

requestSizeBytes - the size of the incoming UPDATE payload (partial update request)

resultSizeBytes - the size of the compressed result value (full record after applying partial update)

largeResultThreshold - results larger than this are tracked in the heavy key map

Returns:

an immutable report snapshot, or null if no report is due