Package com.linkedin.davinci.consumer

Class VeniceAfterImageConsumerImpl<K,V>

java.lang.Object

com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl<K,V>

com.linkedin.davinci.consumer.VeniceAfterImageConsumerImpl<K,V>

All Implemented Interfaces:

VeniceChangelogConsumer<K,V>

Direct Known Subclasses:

LocalBootstrappingVeniceChangelogConsumer

public class VeniceAfterImageConsumerImpl<K,V>
extends VeniceChangelogConsumerImpl<K,V>

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl

VeniceChangelogConsumerImpl.HeartbeatReporterThread

Field Summary

Fields

Modifier and Type	Field	Description
protected long	versionSwapDetectionIntervalTimeInMs

Fields inherited from class com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl

changeCaptureStats, changelogClientConfig, chunkAssembler, compressorFactory, compressorMap, currentValuePayloadSize, currentVersionHighWatermarks, currentVersionLastHeartbeat, heartbeatReporterThread, keyDeserializer, NO_OP_COMPRESSOR, partitionCount, partitionToBootstrapState, partitionToDeleteMessageCount, partitionToPutMessageCount, pubSubConsumer, pubSubTopicRepository, replicationMetadataSchemaRepository, rmdDeserializerCache, schemaReader, specificValueClass, startTimestamp, storeDeserializerCache, storeName, storeRepository, subscribeTime, userEventChunkingAdapter

Constructor Summary

Constructors

Modifier	Constructor	Description
	VeniceAfterImageConsumerImpl(ChangelogClientConfig changelogClientConfig, PubSubConsumerAdapter consumer)
protected	VeniceAfterImageConsumerImpl(ChangelogClientConfig changelogClientConfig, PubSubConsumerAdapter consumer, Lazy<VeniceChangelogConsumerImpl<K,V>> seekConsumer)

Method Summary

Modifier and Type	Method	Description
protected java.util.concurrent.CompletableFuture<java.lang.Void>	internalSeek(java.util.Set<java.lang.Integer> partitions, PubSubTopic targetTopic, com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl.SeekFunction seekAction)
java.util.Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>>	poll(long timeoutInMs)	Polling function to get any available messages from the underlying system for all partitions subscribed.
java.util.concurrent.CompletableFuture<java.lang.Void>	seekToEndOfPush(java.util.Set<java.lang.Integer> partitions)	Seek to the end of the last push for a given set of partitions.
java.util.concurrent.CompletableFuture<java.lang.Void>	seekToTail(java.util.Set<java.lang.Integer> partitions)	Seek to the end of events which have been transmitted to Venice and start consuming new events.
java.util.concurrent.CompletableFuture<java.lang.Void>	seekToTimestamps(java.util.Map<java.lang.Integer,java.lang.Long> timestamps)	Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture.
java.util.concurrent.CompletableFuture<java.lang.Void>	subscribe(java.util.Set<java.lang.Integer> partitions)	Subscribe a set of partitions for a store to this VeniceChangelogConsumer.

Methods inherited from class com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl

close, convertPubSubMessageToPubSubChangeEventMessage, extractOffsetVectorFromMessage, getChangeCaptureStats, getChangelogClientConfig, getHeartbeatReporterThread, getLatestCoordinate, getPartitionCount, getPubSubConsumer, getSubscribeTime, getTopicAssignment, getTopicPartition, getVersionCompressor, handleControlMessage, handleVersionSwapControlMessage, internalPoll, internalPoll, internalSeekToTail, internalSeekToTimestamps, internalSubscribe, isCaughtUp, pause, pause, processRecordBytes, resume, resume, seekToBeginningOfPush, seekToBeginningOfPush, seekToCheckpoint, seekToEndOfPush, seekToTail, seekToTimestamp, setStoreRepository, subscribeAll, switchToNewTopic, unsubscribe, unsubscribeAll

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

versionSwapDetectionIntervalTimeInMs

protected long versionSwapDetectionIntervalTimeInMs

Constructor Detail

VeniceAfterImageConsumerImpl

public VeniceAfterImageConsumerImpl(ChangelogClientConfig changelogClientConfig,
                                    PubSubConsumerAdapter consumer)

VeniceAfterImageConsumerImpl

protected VeniceAfterImageConsumerImpl(ChangelogClientConfig changelogClientConfig,
                                       PubSubConsumerAdapter consumer,
                                       Lazy<VeniceChangelogConsumerImpl<K,V>> seekConsumer)

Method Detail

poll

public java.util.Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>> poll(long timeoutInMs)

Description copied from interface: VeniceChangelogConsumer

Polling function to get any available messages from the underlying system for all partitions subscribed.

Specified by:

poll in interface VeniceChangelogConsumer<K,V>

Overrides:

poll in class VeniceChangelogConsumerImpl<K,V>

Parameters:

timeoutInMs - The maximum time to block/wait in between two polling requests (must not be greater than Long.MAX_VALUE milliseconds)

Returns:

a collection of messages since the last fetch for the subscribed list of topic partitions

seekToTimestamps

public java.util.concurrent.CompletableFuture<java.lang.Void> seekToTimestamps(java.util.Map<java.lang.Integer,java.lang.Long> timestamps)

Description copied from interface: VeniceChangelogConsumer

Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture. Note, this API can only be used to seek on nearline data applied to the current serving version in Venice. This will not seek on data transmitted via Batch Push. If the provided timestamp is lower than the earliest timestamp on a given stream, the earliest event will be returned. THIS WILL NOT SEEK TO DATA WHICH WAS APPLIED ON A PREVIOUS VERSION. You should never seek back in time to a timestamp which is smaller than the current time - rewindTimeInSeconds configured in the hybrid settings for this Venice store. The timestamp passed to this function should be associated to timestamps processed by this interface. The timestamp returned by {@link PubSubMessage.getPubSubMessageTime()} refers to the time when Venice processed the event, and calls to this method will seek based on that sequence of events. Note: it bears no relation to timestamps provided by upstream producers when writing to Venice where a user may optionally provide a timestamp at time of producing a record.

Specified by:

seekToTimestamps in interface VeniceChangelogConsumer<K,V>

Overrides:

seekToTimestamps in class VeniceChangelogConsumerImpl<K,V>

Parameters:

timestamps - a map keyed by a partition ID, and the timestamp checkpoints to seek for each partition.

Returns:

subscribe

public java.util.concurrent.CompletableFuture<java.lang.Void> subscribe(java.util.Set<java.lang.Integer> partitions)

Description copied from interface: VeniceChangelogConsumer

Subscribe a set of partitions for a store to this VeniceChangelogConsumer. The VeniceChangelogConsumer should try to consume messages from all partitions that are subscribed to it.

Specified by:

subscribe in interface VeniceChangelogConsumer<K,V>

Overrides:

subscribe in class VeniceChangelogConsumerImpl<K,V>

Parameters:

partitions - the set of partition to subscribe and consume

Returns:

a future which completes when the partitions are ready to be consumed data

seekToTail

public java.util.concurrent.CompletableFuture<java.lang.Void> seekToTail(java.util.Set<java.lang.Integer> partitions)

Description copied from interface: VeniceChangelogConsumer

Seek to the end of events which have been transmitted to Venice and start consuming new events. This will ONLY consume events transmitted via nearline and incremental push. It will not read batch push data.

Specified by:

seekToTail in interface VeniceChangelogConsumer<K,V>

Overrides:

seekToTail in class VeniceChangelogConsumerImpl<K,V>

Parameters:

partitions - the set of partitions to seek with

Returns:

a future which completes when the operation has succeeded for all partitions.

seekToEndOfPush

public java.util.concurrent.CompletableFuture<java.lang.Void> seekToEndOfPush(java.util.Set<java.lang.Integer> partitions)

Description copied from interface: VeniceChangelogConsumer

Seek to the end of the last push for a given set of partitions. This instructs the consumer to begin consuming events which are transmitted to Venice following the last batch push.

Specified by:

seekToEndOfPush in interface VeniceChangelogConsumer<K,V>

Overrides:

seekToEndOfPush in class VeniceChangelogConsumerImpl<K,V>

Parameters:

partitions - the set of partitions to seek with

Returns:

a future which completes when the operation has succeeded for all partitions.

internalSeek

protected java.util.concurrent.CompletableFuture<java.lang.Void> internalSeek(java.util.Set<java.lang.Integer> partitions,
                                                                              PubSubTopic targetTopic,
                                                                              com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl.SeekFunction seekAction)

Overrides:

internalSeek in class VeniceChangelogConsumerImpl<K,V>