Package com.linkedin.davinci.validation

Class PartitionTracker

java.lang.Object

com.linkedin.davinci.validation.PartitionTracker

public class PartitionTracker
extends java.lang.Object

This class maintains state about all the upstream producers for a given partition. It keeps track of the last segment, last sequence number and incrementally computed checksum for each producer (identified by a producer GUID). This class is thread safe. Locking is at the granularity of producers. Multiple threads can process records from the same partition concurrently. This class also encapsulates the capability to clear expired state, in the functions which take in the maxAgeInMs parameter: - clearExpiredStateAndUpdateOffsetRecord(OffsetRecord, long) - setPartitionState(OffsetRecord, long)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PartitionTracker.DIVErrorMetricCallback

Constructor Summary

Constructors

Constructor	Description
PartitionTracker(java.lang.String topicName, int partition)

Method Summary

Modifier and Type	Method	Description
void	checkMissingMessage(PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord, java.util.Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback, long kafkaLogCompactionDelayInMs)
void	cloneProducerStates(PartitionTracker destProducerTracker)
int	getPartition()
void	setPartitionState(OffsetRecord offsetRecord, long maxAgeInMs)
java.lang.String	toString()
void	updateOffsetRecord(OffsetRecord offsetRecord)
void	validateMessage(PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord, boolean endOfPushReceived, Lazy<java.lang.Boolean> tolerateMissingMsgs)	Ensures the integrity of the data by maintaining state about all of the data produced by a specific upstream producer: 1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PartitionTracker

public PartitionTracker(java.lang.String topicName,
                        int partition)

Method Detail

getPartition

public int getPartition()

toString

public final java.lang.String toString()

Overrides:

toString in class java.lang.Object

setPartitionState

public void setPartitionState(OffsetRecord offsetRecord,
                              long maxAgeInMs)

cloneProducerStates

public void cloneProducerStates(PartitionTracker destProducerTracker)

updateOffsetRecord

public void updateOffsetRecord(OffsetRecord offsetRecord)

validateMessage

public void validateMessage(PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord,
                            boolean endOfPushReceived,
                            Lazy<java.lang.Boolean> tolerateMissingMsgs)
                     throws DataValidationException

Ensures the integrity of the data by maintaining state about all of the data produced by a specific upstream producer: 1. Segment, which should be equal or greater to the previous segment. 2. Sequence number, which should be exactly one greater than the previous sequence number. 3. Checksum, which is computed incrementally until the end of a segment.

Parameters:

consumerRecord -

Throws:

DataValidationException

checkMissingMessage

public void checkMissingMessage(PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord,
                                java.util.Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback,
                                long kafkaLogCompactionDelayInMs)
                         throws DataValidationException

Throws:

DataValidationException