Package com.linkedin.davinci.validation

Class PartitionTracker

java.lang.Object

com.linkedin.davinci.validation.PartitionTracker

public class PartitionTracker
extends Object

This class maintains state about all the upstream producers for a given partition. It keeps track of the last segment, last sequence number and incrementally computed checksum for each producer (identified by a producer GUID).

This class is thread safe. Locking is at the granularity of producers. Multiple threads can process records from the same partition concurrently.

This class also encapsulates the capability to clear expired state, in the functions which take in the maxAgeInMs parameter:

- clearExpiredStateAndUpdateOffsetRecord(TopicType, OffsetRecord, long) - setPartitionState(TopicType, OffsetRecord, long)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PartitionTracker.DIVErrorMetricCallback
static class	PartitionTracker.TopicType

Field Summary

Fields

Modifier and Type	Field	Description
static final PartitionTracker.TopicType	VERSION_TOPIC

Constructor Summary

Constructors

Constructor	Description
PartitionTracker(String topicName, int partition)

Method Summary

Modifier and Type	Method	Description
void	checkMissingMessage(PartitionTracker.TopicType type, DefaultPubSubMessage consumerRecord, Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback, long kafkaLogCompactionDelayInMs)
void	cloneProducerStates(PartitionTracker destProducerTracker, String brokerUrl)	Clone both vtSegment and rtSegment to the destination PartitionTracker.
long	getLatestConsumedVtOffset()
int	getPartition()
Map<CharSequence,ProducerPartitionState>	getPartitionStates(PartitionTracker.TopicType type)
void	removeProducerState(PartitionTracker.TopicType type, GUID guid, OffsetRecord offsetRecord)
void	setPartitionState(PartitionTracker.TopicType type, OffsetRecord offsetRecord, long maxAgeInMs)
void	setProducerState(OffsetRecord offsetRecord, PartitionTracker.TopicType type, GUID guid, ProducerPartitionState state)
final String	toString()
void	updateLatestConsumedVtOffset(long offset)
void	updateOffsetRecord(PartitionTracker.TopicType type, OffsetRecord offsetRecord)
void	validateMessage(PartitionTracker.TopicType type, DefaultPubSubMessage consumerRecord, boolean endOfPushReceived, Lazy<Boolean> tolerateMissingMsgs)	Ensures the integrity of the data by maintaining state about all the data produced by a specific upstream producer:

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

VERSION_TOPIC

public static final PartitionTracker.TopicType VERSION_TOPIC

Constructor Details

PartitionTracker

public PartitionTracker(String topicName,
 int partition)

Method Details

getPartition

public int getPartition()

getLatestConsumedVtOffset

public long getLatestConsumedVtOffset()

updateLatestConsumedVtOffset

public void updateLatestConsumedVtOffset(long offset)

toString

public final String toString()

Overrides:

toString in class Object

setPartitionState

public void setPartitionState(PartitionTracker.TopicType type,
 OffsetRecord offsetRecord,
 long maxAgeInMs)

getPartitionStates

public Map<CharSequence,ProducerPartitionState> getPartitionStates(PartitionTracker.TopicType type)

cloneProducerStates

public void cloneProducerStates(PartitionTracker destProducerTracker,
 String brokerUrl)

Clone both vtSegment and rtSegment to the destination PartitionTracker. May be called concurrently.

setProducerState

public void setProducerState(OffsetRecord offsetRecord,
 PartitionTracker.TopicType type,
 GUID guid,
 ProducerPartitionState state)

updateOffsetRecord

public void updateOffsetRecord(PartitionTracker.TopicType type,
 OffsetRecord offsetRecord)

validateMessage

public void validateMessage(PartitionTracker.TopicType type,
 DefaultPubSubMessage consumerRecord,
 boolean endOfPushReceived,
 Lazy<Boolean> tolerateMissingMsgs)
                     throws DataValidationException

Ensures the integrity of the data by maintaining state about all the data produced by a specific upstream producer:

1. Segment, which should be equal or greater to the previous segment. 2. Sequence number, which should be exactly one greater than the previous sequence number. 3. Checksum, which is computed incrementally until the end of a segment.

Parameters:

consumerRecord - the incoming Kafka message.

Throws:

DataValidationException - if the DIV check failed.

checkMissingMessage

public void checkMissingMessage(PartitionTracker.TopicType type,
 DefaultPubSubMessage consumerRecord,
 Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback,
 long kafkaLogCompactionDelayInMs)
                         throws DataValidationException

Throws:

DataValidationException

removeProducerState

public void removeProducerState(PartitionTracker.TopicType type,
 GUID guid,
 OffsetRecord offsetRecord)