Package com.linkedin.venice.spark.consistency

Class LilyPadUtils

java.lang.Object

com.linkedin.venice.spark.consistency.LilyPadUtils

public final class LilyPadUtils
extends Object

Core logic for the lily-pad VT consistency algorithm.

Given per-key record snapshots from two DCs, identifies keys where the two DCs disagree despite both having had full information. This class is generic over the position type and has no PubSub dependencies. See LilyPadSnapshotBuilder for building snapshots from PubSub.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	LilyPadUtils.Inconsistency<T extends Comparable<T>>	A single detected VT inconsistency between two DCs for one key.
static enum	LilyPadUtils.InconsistencyType
static class	LilyPadUtils.KeyRecord<T extends Comparable<T>>	Per-key snapshot of a single VT record, capturing the value, the upstream position vector that the leader used for its DCR decision, the global high-watermark at that moment, and the logical timestamp used by DCR to pick the winner.
static class	LilyPadUtils.Snapshot<T extends Comparable<T>>	The per-key record history plus the final partition high-watermark.

Method Summary

Modifier and Type	Method	Description
static <T extends Comparable<T>> List<LilyPadUtils.Inconsistency<T>>	findInconsistencies(LilyPadUtils.Snapshot<T> dc0Snapshot, LilyPadUtils.Snapshot<T> dc1Snapshot)
static <T extends Comparable<T>> LilyPadUtils.Inconsistency<T>	findInconsistencyForKey(long keyHash, List<LilyPadUtils.KeyRecord<T>> dc0History, List<LilyPadUtils.KeyRecord<T>> dc1History, List<T> dc0PartitionHighWatermark, List<T> dc1PartitionHighWatermark, long[] pairStats)	Runs the lily-pad comparison for a single key.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

findInconsistencies

public static <T extends Comparable<T>>
List<LilyPadUtils.Inconsistency<T>> findInconsistencies(LilyPadUtils.Snapshot<T> dc0Snapshot,
 LilyPadUtils.Snapshot<T> dc1Snapshot)

findInconsistencyForKey

public static <T extends Comparable<T>>
LilyPadUtils.Inconsistency<T> findInconsistencyForKey(long keyHash,
 List<LilyPadUtils.KeyRecord<T>> dc0History,
 List<LilyPadUtils.KeyRecord<T>> dc1History,
 List<T> dc0PartitionHighWatermark,
 List<T> dc1PartitionHighWatermark,
 long[] pairStats)

Runs the lily-pad comparison for a single key. Returns an LilyPadUtils.Inconsistency if one is detected, or null if the key is consistent (or not yet comparable due to replication lag).

Parameters:

keyHash - 64-bit hash of the key being compared

dc0History - ordered record history for this key in DC-0 (may be empty)

dc1History - ordered record history for this key in DC-1 (may be empty)

dc0PartitionHighWatermark - final partition-level high watermark from DC-0's snapshot

dc1PartitionHighWatermark - final partition-level high watermark from DC-1's snapshot

Returns:

an LilyPadUtils.Inconsistency if detected, or null if consistent