Package com.linkedin.davinci.consumer

Class LocalBootstrappingVeniceChangelogConsumer<K,V>

java.lang.Object

com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl<K,V>

com.linkedin.davinci.consumer.VeniceAfterImageConsumerImpl<K,V>

com.linkedin.davinci.consumer.LocalBootstrappingVeniceChangelogConsumer<K,V>

Type Parameters:

K -

V -

All Implemented Interfaces:

BootstrappingVeniceChangelogConsumer<K,V>, VeniceChangelogConsumer<K,V>

public class LocalBootstrappingVeniceChangelogConsumer<K,V>
extends VeniceAfterImageConsumerImpl<K,V>

This is a wrapper class on top of InternalBootstrappingVeniceChangelogConsumerImpl. This confines the usage of this class for clients to the methods exposed on the interface. This is meant to prevent users from doing seek() calls which would render the local state inconsistent.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl

VeniceChangelogConsumerImpl.HeartbeatReporterThread

Field Summary

Fields inherited from class com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl

changeCaptureStats, changelogClientConfig, chunkAssembler, compressorFactory, compressorMap, consumerSequenceIdGeneratorMap, consumerSequenceIdStartingValue, currentVersionHighWatermarks, currentVersionLastHeartbeat, heartbeatReporterThread, isSubscribed, keyDeserializer, NO_OP_COMPRESSOR, partitionToBootstrapState, partitionToDeleteMessageCount, partitionToPutMessageCount, pubSubConsumer, pubSubContext, pubSubMessageDeserializer, pubSubPositionDeserializer, pubSubTopicRepository, replicationMetadataSchemaRepository, rmdDeserializerCache, schemaReader, seekExecutorService, specificValueClass, startTimestamp, storeDeserializerCache, storeName, storeRepository, subscribeTime, subscriptionLock, userEventChunkingAdapter

Constructor Summary

Constructors

Constructor	Description
LocalBootstrappingVeniceChangelogConsumer(ChangelogClientConfig changelogClientConfig, PubSubConsumerAdapter pubSubConsumer, PubSubMessageDeserializer pubSubMessageDeserializer, String consumerId, VeniceChangelogConsumerClientFactory veniceChangelogConsumerClientFactory)

Method Summary

Modifier and Type	Method	Description
protected boolean	handleVersionSwapControlMessage(ControlMessage controlMessage, PubSubTopicPartition pubSubTopicPartition, String topicSuffix, Integer upstreamPartition)
protected Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>>	internalPoll(long timeoutInMs, String topicSuffix)
protected <T> T	processRecordBytes(ByteBuffer decompressedBytes, T deserializedValue, byte[] key, ByteBuffer value, PubSubTopicPartition partition, int readerSchemaId, PubSubPosition recordOffset)
CompletableFuture<Void>	seekWithBootStrap(Set<Integer> partitions)
CompletableFuture<Void>	start()
CompletableFuture<Void>	start(Set<Integer> partitions)	Start performs both a topic subscription and catch up.
void	stop()

Methods inherited from class com.linkedin.davinci.consumer.VeniceAfterImageConsumerImpl

adjustSeekCheckPointsBasedOnHeartbeats, close, handleVersionSwapFailure, internalSeek, internalSeekToEndOfPush, poll, seekToEndOfPush, seekToTail, seekToTimestamps, setStoreRepository, subscribe, subscribed

Methods inherited from class com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl

convertPubSubMessageToPubSubChangeEventMessage, extractOffsetVectorFromMessage, getChangeCaptureStats, getChangelogClientConfig, getCurrentServingVersionTopic, getHeartbeatReporterThread, getLatestCoordinate, getNextConsumerSequenceId, getPartitionCount, getPartitionListToSubscribe, getSubscribeTime, getTopicAssignment, getTopicPartition, getVersionCompressor, handleControlMessage, internalPoll, internalSeekToTail, internalSeekToTimestamps, internalSeekToTimestamps, internalSubscribe, isCaughtUp, pause, pause, resume, resume, seekToBeginningOfPush, seekToBeginningOfPush, seekToCheckpoint, seekToEndOfPush, seekToTail, seekToTimestamp, subscribeAll, switchToNewTopic, synchronousSeek, synchronousSeekToCheckpoint, unsubscribe, unsubscribeAll

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.davinci.consumer.BootstrappingVeniceChangelogConsumer

isCaughtUp, poll

Constructor Details

LocalBootstrappingVeniceChangelogConsumer

public LocalBootstrappingVeniceChangelogConsumer(ChangelogClientConfig changelogClientConfig,
 PubSubConsumerAdapter pubSubConsumer,
 PubSubMessageDeserializer pubSubMessageDeserializer,
 String consumerId,
 VeniceChangelogConsumerClientFactory veniceChangelogConsumerClientFactory)

Method Details

handleVersionSwapControlMessage

protected boolean handleVersionSwapControlMessage(ControlMessage controlMessage,
 PubSubTopicPartition pubSubTopicPartition,
 String topicSuffix,
 Integer upstreamPartition)

Overrides:

handleVersionSwapControlMessage in class VeniceChangelogConsumerImpl<K,V>

internalPoll

protected Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>> internalPoll(long timeoutInMs,
 String topicSuffix)

Overrides:

internalPoll in class VeniceChangelogConsumerImpl<K,V>

processRecordBytes

protected <T> T processRecordBytes(ByteBuffer decompressedBytes,
 T deserializedValue,
 byte[] key,
 ByteBuffer value,
 PubSubTopicPartition partition,
 int readerSchemaId,
 PubSubPosition recordOffset)

Overrides:

processRecordBytes in class VeniceChangelogConsumerImpl<K,V>

seekWithBootStrap

public CompletableFuture<Void> seekWithBootStrap(Set<Integer> partitions)

start

public CompletableFuture<Void> start(Set<Integer> partitions)

Description copied from interface: BootstrappingVeniceChangelogConsumer

Start performs both a topic subscription and catch up. The client will look at the latest offset in the server and sync bootstrap data up to that point in changes. Once that is done for all partitions, the future will complete. NOTE: This future may take some time to complete depending on how much data needs to be ingested in order to catch up with the time that this client started. NOTE: In the experimental client, the future will complete when there is at least one message to be polled. We don't wait for all partitions to catch up, as loading every message into a buffer will result in an Out Of Memory error. Instead, use the BootstrappingVeniceChangelogConsumer.isCaughtUp() method to determine once all subscribed partitions have caught up. NOTE: In the experimental client, if you pass in an empty set, it will subscribe to all partitions for the store

Specified by:

start in interface BootstrappingVeniceChangelogConsumer<K,V>

Parameters:

partitions - which partition id's to catch up with

Returns:

a future that completes once catch up is complete for all passed in partitions.

start

public CompletableFuture<Void> start()

Specified by:

start in interface BootstrappingVeniceChangelogConsumer<K,V>

stop

public void stop()
          throws Exception

Specified by:

stop in interface BootstrappingVeniceChangelogConsumer<K,V>

Throws:

Exception