Package com.linkedin.venice.spark.consistency

Class LilyPadUtils

java.lang.Object

com.linkedin.venice.spark.consistency.LilyPadUtils

public final class LilyPadUtils
extends Object

Core logic for the lily-pad VT consistency algorithm.

Given per-key record snapshots from two DCs, identifies keys where the two DCs disagree despite both having had full information. This class is generic over the position type and has no PubSub dependencies. See LilyPadSnapshotBuilder for building snapshots from PubSub.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	LilyPadUtils.Inconsistency<T extends Comparable<T>>	A single detected VT inconsistency between two DCs for one key.
static enum	LilyPadUtils.InconsistencyType
static class	LilyPadUtils.KeyRecord<T extends Comparable<T>>	Per-key snapshot of a single VT record, capturing the value, the upstream position vector that the leader used for its DCR decision, the global high-watermark at that moment, and the logical timestamp used by DCR to pick the winner.
static class	LilyPadUtils.Snapshot<T extends Comparable<T>>	The per-key record history plus the final partition high-watermark.

Method Summary

Modifier and Type	Method	Description
static <T extends Comparable<T>> List<LilyPadUtils.Inconsistency<T>>	findInconsistencies(LilyPadUtils.Snapshot<T> dc0Snapshot, LilyPadUtils.Snapshot<T> dc1Snapshot)	Runs the lily-pad algorithm over two pre-built snapshots and returns all detected inconsistencies.
static <T extends Comparable<T>> LilyPadUtils.Inconsistency<T>	findInconsistencyForKey(long keyHash, List<LilyPadUtils.KeyRecord<T>> dc0History, List<LilyPadUtils.KeyRecord<T>> dc1History, List<T> dc0PartitionHighWatermark, List<T> dc1PartitionHighWatermark)	Runs the lily-pad comparison for a single key.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

findInconsistencies

public static <T extends Comparable<T>>
List<LilyPadUtils.Inconsistency<T>> findInconsistencies(LilyPadUtils.Snapshot<T> dc0Snapshot,
 LilyPadUtils.Snapshot<T> dc1Snapshot)

Runs the lily-pad algorithm over two pre-built snapshots and returns all detected inconsistencies. An empty list means the two DCs agree on every comparable key.

Two records are comparable when each DC's global high-watermark covers every per-key position the other DC considered — meaning both leaders had full information when writing. Non-comparable pairs are skipped (replication lag, not a real inconsistency).

Parameters:

dc0Snapshot - snapshot built by LilyPadSnapshotBuilder.buildSnapshot(com.linkedin.venice.vpj.pubsub.input.PubSubSplitIterator, com.linkedin.venice.pubsub.manager.TopicManager, com.linkedin.venice.pubsub.PubSubPositionDeserializer, int) for DC-0

dc1Snapshot - snapshot built by LilyPadSnapshotBuilder.buildSnapshot(com.linkedin.venice.vpj.pubsub.input.PubSubSplitIterator, com.linkedin.venice.pubsub.manager.TopicManager, com.linkedin.venice.pubsub.PubSubPositionDeserializer, int) for DC-1

Returns:

list of LilyPadUtils.Inconsistency objects, one per inconsistent key

findInconsistencyForKey

public static <T extends Comparable<T>>
LilyPadUtils.Inconsistency<T> findInconsistencyForKey(long keyHash,
 List<LilyPadUtils.KeyRecord<T>> dc0History,
 List<LilyPadUtils.KeyRecord<T>> dc1History,
 List<T> dc0PartitionHighWatermark,
 List<T> dc1PartitionHighWatermark)

Runs the lily-pad comparison for a single key. Returns an LilyPadUtils.Inconsistency if one is detected, or null if the key is consistent (or not yet comparable due to replication lag).

Parameters:

keyHash - 64-bit hash of the key being compared

dc0History - ordered record history for this key in DC-0 (may be empty)

dc1History - ordered record history for this key in DC-1 (may be empty)

dc0PartitionHighWatermark - final partition-level high watermark from DC-0's snapshot

dc1PartitionHighWatermark - final partition-level high watermark from DC-1's snapshot

Returns:

an LilyPadUtils.Inconsistency if detected, or null if consistent