Package com.linkedin.venice.spark.utils
Class PubSubPositionRowComparator
java.lang.Object
com.linkedin.venice.spark.utils.PubSubPositionRowComparator
- All Implemented Interfaces:
Serializable,Comparator<org.apache.spark.sql.Row>
public class PubSubPositionRowComparator
extends Object
implements Comparator<org.apache.spark.sql.Row>, Serializable
Orders KIF repush rows of the same key by their position in the source topic-partition (older first).
Fast path: when both rows carry a non-negative SparkConstants.OFFSET_COLUMN_NAME,
rows are ordered by that numeric offset, exactly as before. Otherwise (e.g. pub-sub systems whose positions have no
dense numeric offset and report -1), the serialized positions in
SparkConstants.POSITION_COLUMN_NAME are deserialized on the executor and compared
with PubSubPosition.comparePositionOffline(PubSubPosition). If the positions cannot be ordered, a
VeniceException is thrown rather than silently picking an arbitrary record.
The position type registry is rebuilt on the executor from the same job property the PubSub input reader uses
(PUBSUB_TYPE_ID_TO_POSITION_CLASS_NAME_MAP), so no additional configuration is needed.
- See Also:
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionintcompare(org.apache.spark.sql.Row r1, org.apache.spark.sql.Row r2) org.apache.spark.sql.RowselectLatest(List<org.apache.spark.sql.Row> rows) voidsortLatestFirst(List<org.apache.spark.sql.Row> rows) Sortsrowsin place, newest (highest position) first.Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface java.util.Comparator
equals, reversed, thenComparing, thenComparing, thenComparing, thenComparingDouble, thenComparingInt, thenComparingLong
-
Constructor Details
-
PubSubPositionRowComparator
-
-
Method Details
-
compare
public int compare(org.apache.spark.sql.Row r1, org.apache.spark.sql.Row r2) - Specified by:
comparein interfaceComparator<org.apache.spark.sql.Row>
-
selectLatest
- Returns:
- the newest row (highest position) of
rows, or null ifrowsis empty. On ties, the first such row is returned. Each position is deserialized at most once.
-
sortLatestFirst
Sortsrowsin place, newest (highest position) first. The sort is stable and each position is deserialized at most once.
-