Class PartitionTracker

  • public class PartitionTracker
    extends java.lang.Object
    This class maintains state about all the upstream producers for a given partition. It keeps track of the last segment, last sequence number and incrementally computed checksum for each producer (identified by a producer GUID). This class is thread safe. Locking is at the granularity of producers. Multiple threads can process records from the same partition concurrently. This class also encapsulates the capability to clear expired state, in the functions which take in the maxAgeInMs parameter: - clearExpiredStateAndUpdateOffsetRecord(OffsetRecord, long) - setPartitionState(OffsetRecord, long)
    • Constructor Detail

      • PartitionTracker

        public PartitionTracker​(java.lang.String topicName,
                                int partition)
    • Method Detail

      • getPartition

        public int getPartition()
      • toString

        public final java.lang.String toString()
        toString in class java.lang.Object
      • setPartitionState

        public void setPartitionState​(OffsetRecord offsetRecord,
                                      long maxAgeInMs)
      • cloneProducerStates

        public void cloneProducerStates​(PartitionTracker destProducerTracker)
      • updateOffsetRecord

        public void updateOffsetRecord​(OffsetRecord offsetRecord)
      • validateMessage

        public void validateMessage​(PubSubMessage<KafkaKey,​KafkaMessageEnvelope,​java.lang.Long> consumerRecord,
                                    boolean endOfPushReceived,
                                    Lazy<java.lang.Boolean> tolerateMissingMsgs)
                             throws DataValidationException
        Ensures the integrity of the data by maintaining state about all of the data produced by a specific upstream producer: 1. Segment, which should be equal or greater to the previous segment. 2. Sequence number, which should be exactly one greater than the previous sequence number. 3. Checksum, which is computed incrementally until the end of a segment.
        consumerRecord -