Class PubSubPositionRowComparator

java.lang.Object
com.linkedin.venice.spark.utils.PubSubPositionRowComparator
All Implemented Interfaces:
Serializable, Comparator<org.apache.spark.sql.Row>

public class PubSubPositionRowComparator extends Object implements Comparator<org.apache.spark.sql.Row>, Serializable
Orders KIF repush rows of the same key by their position in the source topic-partition (older first).

Fast path: when both rows carry a non-negative SparkConstants.OFFSET_COLUMN_NAME, rows are ordered by that numeric offset, exactly as before. Otherwise (e.g. pub-sub systems whose positions have no dense numeric offset and report -1), the serialized positions in SparkConstants.POSITION_COLUMN_NAME are deserialized on the executor and compared with PubSubPosition.comparePositionOffline(PubSubPosition). If the positions cannot be ordered, a VeniceException is thrown rather than silently picking an arbitrary record.

The position type registry is rebuilt on the executor from the same job property the PubSub input reader uses (PUBSUB_TYPE_ID_TO_POSITION_CLASS_NAME_MAP), so no additional configuration is needed.

See Also:
  • Constructor Details

    • PubSubPositionRowComparator

      public PubSubPositionRowComparator(VeniceProperties jobProps)
  • Method Details

    • compare

      public int compare(org.apache.spark.sql.Row r1, org.apache.spark.sql.Row r2)
      Specified by:
      compare in interface Comparator<org.apache.spark.sql.Row>
    • selectLatest

      public org.apache.spark.sql.Row selectLatest(List<org.apache.spark.sql.Row> rows)
      Returns:
      the newest row (highest position) of rows, or null if rows is empty. On ties, the first such row is returned. Each position is deserialized at most once.
    • sortLatestFirst

      public void sortLatestFirst(List<org.apache.spark.sql.Row> rows)
      Sorts rows in place, newest (highest position) first. The sort is stable and each position is deserialized at most once.