Package com.linkedin.venice.utils.consistency

Class DiffValidationUtils

java.lang.Object
com.linkedin.venice.utils.consistency.DiffValidationUtils
public final class DiffValidationUtils extends Object

  • Method Summary

    Modifier and Type
    Method
    Description
    static boolean
    doRecordsDiverge(String firstValueChecksum, String secondValueChecksum, List<Long> firstValuePartitionOffsetWatermark, List<Long> secondValuePartitionOffsetWatermark, List<Long> firstValueOffsetVector, List<Long> secondValueOffsetVector)
    This method determines that if given the values for two equal keys in a snapshot of venice data, have the records diverged.
    static <T extends Comparable<T>>
    boolean
    hasOffsetAdvanced(List<T> baseOffset, List<T> advancedOffset)
    Checks to see if an offset vector has advanced completely beyond some base offset vector or not.
    static <T extends Comparable<T>>
    boolean
    isRecordMissing(List<T> existingValueOffsetVector, List<T> nonExistentValuePartitionOffsetWatermark)
    This method determines if a given record in a snapshot has been lost.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • isRecordMissing

      public static <T extends Comparable<T>> boolean isRecordMissing(List<T> existingValueOffsetVector, List<T> nonExistentValuePartitionOffsetWatermark)
      This method determines if a given record in a snapshot has been lost. It determines this by looking at the records ReplicationMetadata and comparing it to the highwatermark of the colo which did not have this record. If the colo which does not have this record has advanced beyond the offset vector held by the individual record, then we know that this record has been passed by. This function should not be used on any record which holds a tombstone (as the colo which is farther ahead will have purged out a record which currently holds a tombstone).
      Parameters:
      existingValueOffsetVector -
      nonExistentValuePartitionOffsetWatermark -
      Returns:
      True if the record seems to be missing.

    • doRecordsDiverge

      public static boolean doRecordsDiverge(String firstValueChecksum, String secondValueChecksum, List<Long> firstValuePartitionOffsetWatermark, List<Long> secondValuePartitionOffsetWatermark, List<Long> firstValueOffsetVector, List<Long> secondValueOffsetVector)
      This method determines that if given the values for two equal keys in a snapshot of venice data, have the records diverged. It does this, by determining if each colo has received all data which is pertinent to coming to the current resolution held by each snapshot. In other words, does each colo have all the same information to come to the same conclusion, and do these conclusions diverge? This method DOES NOT catch divergence for 'missing' records. Missing records should be determined with {

    • hasOffsetAdvanced

      public static <T extends Comparable<T>> boolean hasOffsetAdvanced(List<T> baseOffset, List<T> advancedOffset)
      Checks to see if an offset vector has advanced completely beyond some base offset vector or not. Index = region ID. Null entries represent regions not yet seen and are handled as follows:
      • null in base → trivially covered (region not seen yet), skipped
      • null in advanced but non-null in base → not yet covered, returns false
      Parameters:
      baseOffset - The vector to compare against, indexed by region ID.
      advancedOffset - The vector that should be advanced along, indexed by region ID.
      Returns:
      True if the advancedOffset vector has grown beyond the baseOffset.