Package com.linkedin.davinci.validation

Class PartitionTracker

java.lang.Object

com.linkedin.davinci.validation.PartitionTracker

public class PartitionTracker
extends Object

This class maintains state about all the upstream producers for a given partition. It keeps track of the last segment, last sequence number and incrementally computed checksum for each producer (identified by a producer GUID).

This class is thread safe. Locking is at the granularity of producers. Multiple threads can process records from the same partition concurrently.

This class also encapsulates the capability to clear expired state, in the functions which take in the maxAgeInMs parameter:

- clearExpiredStateAndUpdateOffsetRecord(TopicType, OffsetRecord, long) - setPartitionState(TopicType, OffsetRecord, long)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PartitionTracker.DIVErrorMetricCallback
static class	PartitionTracker.TopicType

Field Summary

Fields

Modifier and Type	Field	Description
static final PartitionTracker.TopicType	VERSION_TOPIC

Constructor Summary

Constructors

Constructor	Description
PartitionTracker(String topicName, int partition, PubSubPositionDeserializer pubSubPositionDeserializer)

Method Summary

Modifier and Type	Method	Description
void	checkMissingMessage(PartitionTracker.TopicType type, DefaultPubSubMessage consumerRecord, Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback, long kafkaLogCompactionDelayInMs)
void	clearSegments(PartitionTracker.TopicType type)
void	cloneRtProducerStates(PartitionTracker destProducerTracker, String brokerUrl)	Clone the rtSegments to the destination PartitionTracker.
void	cloneVtProducerStates(PartitionTracker destProducerTracker)	Clone the vtSegments and LCVP to the destination PartitionTracker.
PubSubPosition	getLatestConsumedVtPosition()
int	getPartition()
Map<CharSequence,ProducerPartitionState>	getPartitionStates(PartitionTracker.TopicType type)
void	removeProducerState(PartitionTracker.TopicType type, GUID guid, OffsetRecord offsetRecord)
void	setPartitionState(PartitionTracker.TopicType type, OffsetRecord offsetRecord, long maxAgeInMs)
void	setPartitionState(PartitionTracker.TopicType type, Map<CharSequence,ProducerPartitionState> producerPartitionStateMap, long minimumRequiredRecordProducerTimestamp)
void	setProducerState(OffsetRecord offsetRecord, PartitionTracker.TopicType type, GUID guid, ProducerPartitionState state)
final String	toString()
void	updateLatestConsumedVtPosition(PubSubPosition vtPosition)
void	updateOffsetRecord(PartitionTracker.TopicType type, OffsetRecord offsetRecord)
void	validateMessage(PartitionTracker.TopicType type, DefaultPubSubMessage consumerRecord, boolean endOfPushReceived, Lazy<Boolean> tolerateMissingMsgs)	Ensures the integrity of the data by maintaining state about all the data produced by a specific upstream producer:

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

VERSION_TOPIC

public static final PartitionTracker.TopicType VERSION_TOPIC

Constructor Details

PartitionTracker

public PartitionTracker(String topicName,
 int partition,
 PubSubPositionDeserializer pubSubPositionDeserializer)

Method Details

getPartition

public int getPartition()

getLatestConsumedVtPosition

public PubSubPosition getLatestConsumedVtPosition()

updateLatestConsumedVtPosition

public void updateLatestConsumedVtPosition(PubSubPosition vtPosition)

toString

public final String toString()

Overrides:

toString in class Object

clearSegments

public void clearSegments(PartitionTracker.TopicType type)

setPartitionState

public void setPartitionState(PartitionTracker.TopicType type,
 OffsetRecord offsetRecord,
 long maxAgeInMs)

setPartitionState

public void setPartitionState(PartitionTracker.TopicType type,
 Map<CharSequence,ProducerPartitionState> producerPartitionStateMap,
 long minimumRequiredRecordProducerTimestamp)

getPartitionStates

public Map<CharSequence,ProducerPartitionState> getPartitionStates(PartitionTracker.TopicType type)

cloneVtProducerStates

public void cloneVtProducerStates(PartitionTracker destProducerTracker)

Clone the vtSegments and LCVP to the destination PartitionTracker. May be called concurrently.

cloneRtProducerStates

public void cloneRtProducerStates(PartitionTracker destProducerTracker,
 String brokerUrl)

Clone the rtSegments to the destination PartitionTracker. Filter by brokerUrl. May be called concurrently.

setProducerState

public void setProducerState(OffsetRecord offsetRecord,
 PartitionTracker.TopicType type,
 GUID guid,
 ProducerPartitionState state)

updateOffsetRecord

public void updateOffsetRecord(PartitionTracker.TopicType type,
 OffsetRecord offsetRecord)

validateMessage

public void validateMessage(PartitionTracker.TopicType type,
 DefaultPubSubMessage consumerRecord,
 boolean endOfPushReceived,
 Lazy<Boolean> tolerateMissingMsgs)
                     throws DataValidationException

Ensures the integrity of the data by maintaining state about all the data produced by a specific upstream producer:

1. Segment, which should be equal or greater to the previous segment. 2. Sequence number, which should be exactly one greater than the previous sequence number. 3. Checksum, which is computed incrementally until the end of a segment.

Parameters:

consumerRecord - the incoming Kafka message.

Throws:

DataValidationException - if the DIV check failed.

checkMissingMessage

public void checkMissingMessage(PartitionTracker.TopicType type,
 DefaultPubSubMessage consumerRecord,
 Optional<PartitionTracker.DIVErrorMetricCallback> errorMetricCallback,
 long kafkaLogCompactionDelayInMs)
                         throws DataValidationException

Throws:

DataValidationException

removeProducerState

public void removeProducerState(PartitionTracker.TopicType type,
 GUID guid,
 OffsetRecord offsetRecord)