Package com.linkedin.davinci.validation

Class PartitionTracker

java.lang.Object

com.linkedin.davinci.validation.PartitionTracker

public class PartitionTracker
extends java.lang.Object

This class maintains state about all the upstream producers for a given partition. It keeps track of the last segment, last sequence number and incrementally computed checksum for each producer (identified by a producer GUID).

This class is thread safe. Locking is at the granularity of producers. Multiple threads can process records from the same partition concurrently.

This class also encapsulates the capability to clear expired state, in the functions which take in the maxAgeInMs parameter:

- clearExpiredStateAndUpdateOffsetRecord(TopicType, OffsetRecord, long) - setPartitionState(TopicType, OffsetRecord, long)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PartitionTracker.DIVErrorMetricCallback
static class	PartitionTracker.TopicType

Field Summary

Fields

Modifier and Type	Field	Description
static PartitionTracker.TopicType	VERSION_TOPIC

Constructor Summary

Constructors

Constructor	Description
PartitionTracker(java.lang.String topicName, int partition)

Method Summary

Modifier and Type	Method	Description
void	checkMissingMessage(PartitionTracker.TopicType type, PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord, java.util.Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback, long kafkaLogCompactionDelayInMs)
void	cloneProducerStates(PartitionTracker destProducerTracker)
int	getPartition()
void	removeProducerState(PartitionTracker.TopicType type, GUID guid, OffsetRecord offsetRecord)
void	setPartitionState(PartitionTracker.TopicType type, OffsetRecord offsetRecord, long maxAgeInMs)
void	setProducerState(OffsetRecord offsetRecord, PartitionTracker.TopicType type, GUID guid, ProducerPartitionState state)
java.lang.String	toString()
void	updateOffsetRecord(PartitionTracker.TopicType type, OffsetRecord offsetRecord)
void	validateMessage(PartitionTracker.TopicType type, PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord, boolean endOfPushReceived, Lazy<java.lang.Boolean> tolerateMissingMsgs)	Ensures the integrity of the data by maintaining state about all the data produced by a specific upstream producer:

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

VERSION_TOPIC

public static final PartitionTracker.TopicType VERSION_TOPIC

Constructor Detail

PartitionTracker

public PartitionTracker(java.lang.String topicName,
                        int partition)

Method Detail

getPartition

public int getPartition()

toString

public final java.lang.String toString()

Overrides:

toString in class java.lang.Object

setPartitionState

public void setPartitionState(PartitionTracker.TopicType type,
                              OffsetRecord offsetRecord,
                              long maxAgeInMs)

cloneProducerStates

public void cloneProducerStates(PartitionTracker destProducerTracker)

setProducerState

public void setProducerState(OffsetRecord offsetRecord,
                             PartitionTracker.TopicType type,
                             GUID guid,
                             ProducerPartitionState state)

updateOffsetRecord

public void updateOffsetRecord(PartitionTracker.TopicType type,
                               OffsetRecord offsetRecord)

validateMessage

public void validateMessage(PartitionTracker.TopicType type,
                            PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord,
                            boolean endOfPushReceived,
                            Lazy<java.lang.Boolean> tolerateMissingMsgs)
                     throws DataValidationException

Ensures the integrity of the data by maintaining state about all the data produced by a specific upstream producer:

1. Segment, which should be equal or greater to the previous segment. 2. Sequence number, which should be exactly one greater than the previous sequence number. 3. Checksum, which is computed incrementally until the end of a segment.

Parameters:

consumerRecord - the incoming Kafka message.

Throws:

DataValidationException - if the DIV check failed.

checkMissingMessage

public void checkMissingMessage(PartitionTracker.TopicType type,
                                PubSubMessage<KafkaKey,KafkaMessageEnvelope,java.lang.Long> consumerRecord,
                                java.util.Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback,
                                long kafkaLogCompactionDelayInMs)
                         throws DataValidationException

Throws:

DataValidationException

removeProducerState

public void removeProducerState(PartitionTracker.TopicType type,
                                GUID guid,
                                OffsetRecord offsetRecord)