Package com.linkedin.venice.listener.profiler

Class KeyPartitionProfiler

java.lang.Object
com.linkedin.venice.listener.profiler.KeyPartitionProfiler
public final class KeyPartitionProfiler extends Object
On-demand profiler that records per-partition request counts and per-partition top-K hot keys for a single Venice store version over a bounded time window.

Two-phase recording

The hot path is split into a warm-up phase and a capture phase to keep per-record cost lock-free even on the hot partition:

  • Phase 1 — warm-up (first WARMUP_FRACTION of the session): every record(byte[], int) call increments the per-partition counter (LongAdder) and the CMS (CountMinSketch backed by AtomicLongArray). Nothing else is touched. Every operation is lock-free.
  • Phase 2 — capture (remaining session): counters + CMS as above, plus a per-partition Set of hashed keys whose CMS estimate has crossed the natural top-K threshold (count × K ≥ partitionTotal). The Set is ConcurrentHashMap.newKeySet() — also lock-free. Long-tail keys never enter the Set; only keys that are at-or-above the 1/K fraction of partition traffic do.

At snapshot time we re-evaluate each Set member against the final partitionTotal / K threshold (filters out keys that briefly crossed early but fell back) and push them through a bounded top-K min-heap. The heap is built once per snapshot — not on the hot path — so the per-update lock-contention problem of an always-on heap is gone.

Why WARMUP_FRACTION = 1/3

A key emerging at fraction τ of the session with frequency p is captured iff p × K × (1 − τ/T) ≥ 1. The worst-case captured key — one that emerges right at the warm-up boundary — needs p ≥ 1 / (K × (1 − w)). Setting w = 1/3 means we still capture keys that are at least 1.5× the natural top-K threshold, which leaves enough fidelity to surface meaningfully-hot late-emerging keys while keeping enough Phase 1 time for the CMS to settle. (w = 1/2 would only catch ≥ 2× keys; w = 1/5 would catch ≥ 1.25× keys but with less CMS warm-up.)

PII safety

Raw keys are never stored. Each key is hashed with murmur3-128 before being passed to the CMS or the per-partition Set.

  • Field Details

    • WARMUP_FRACTION

      public static final double WARMUP_FRACTION
      Fraction of the session spent in Phase 1 (CMS + counters only). At 1/3 the worst- case late-emerging key captured by Phase 2 has frequency ≥ 1.5/K, derived from p_min = 1/(K × (1 − w)). See class Javadoc for the full derivation.
      See Also:

  • Constructor Details

    • KeyPartitionProfiler

      public KeyPartitionProfiler(String storeName, String storeVersion, long startTimeMs, long durationMs, int partitionCount, int maxTopK)

  • Method Details

    • record

      public void record(byte[] keyBytes, int partitionId)
      Hot-path entry. Records a single read for the given key/partition. Every operation is lock-free.

      Phase 1: increments counter and CMS only.
      Phase 2: also adds the key to the per-partition Set iff its CMS estimate already crosses the natural top-K threshold (count × K ≥ partitionTotal).

    • isExpired

      public boolean isExpired()

    • getStoreName

      public String getStoreName()

    • getStoreVersion

      public String getStoreVersion()

    • toJson

      public String toJson()
      Emit the full profile as a single-line JSON object. Convenience for tests; production code should prefer emitJsonTo(org.apache.logging.log4j.Logger, java.lang.String, java.lang.String) which splits the payload across multiple log lines.

      Shape: 

       {
   "storeName": ..., "storeVersion": ...,
   "startTimeMs": ..., "durationMs": ..., "actualDurationMs": ...,
   "totalRequests": ..., "skewFactor": ...,
   "partitionDistribution": [ { "partitionId", "count", "percentage" }, ... ],
   "topKeysByPartition": { "<partitionId>": [ { "keyHash", "estimatedCount",
                                                 "percentageOfPartition" }, ... ] }
 }

    • emitJsonTo

      public void emitJsonTo(org.apache.logging.log4j.Logger logger, String logPrefix, String reason)
      Emit the profile across multiple log lines. One header line carries the session-level metrics + partition distribution; each non-empty partition's top-K is emitted on its own subsequent line, sharing a common sessionId for correlation.

      Splitting avoids single log lines in the hundreds of MB at large topK × partitionCount (some async appenders OOM or truncate at line-length limits).