Package com.linkedin.davinci.consumer

Class VeniceAfterImageConsumerImpl<K,V>

java.lang.Object
com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl<K,V>
com.linkedin.davinci.consumer.VeniceAfterImageConsumerImpl<K,V>
All Implemented Interfaces:
VeniceChangelogConsumer<K,V>
Direct Known Subclasses:
LocalBootstrappingVeniceChangelogConsumer
public class VeniceAfterImageConsumerImpl<K,V> extends VeniceChangelogConsumerImpl<K,V>

  • Field Details

    • versionSwapDetectionIntervalTimeInMs

      protected long versionSwapDetectionIntervalTimeInMs

  • Constructor Details

  • Method Details

    • poll

      public Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>> poll(long timeoutInMs)
      Description copied from interface: VeniceChangelogConsumer
      Polling function to get any available messages from the underlying system for all partitions subscribed.
      Specified by:
      poll in interface VeniceChangelogConsumer<K,V>
      Overrides:
      poll in class VeniceChangelogConsumerImpl<K,V>
      Parameters:
      timeoutInMs - The maximum time to block/wait in between two polling requests (must not be greater than Long.MAX_VALUE milliseconds)
      Returns:
      a collection of messages since the last fetch for the subscribed list of topic partitions

    • seekToTimestamps

      public CompletableFuture<Void> seekToTimestamps(Map<Integer,Long> timestamps)
      Description copied from interface: VeniceChangelogConsumer
      Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture. Note, this API can only be used to seek on nearline data applied to the current serving version in Venice. This will not seek on data transmitted via Batch Push. If the provided timestamp is lower than the earliest timestamp on a given stream, the earliest event will be returned. THIS WILL NOT SEEK TO DATA WHICH WAS APPLIED ON A PREVIOUS VERSION. You should never seek back in time to a timestamp which is smaller than the current time - rewindTimeInSeconds configured in the hybrid settings for this Venice store. The timestamp passed to this function should be associated to timestamps processed by this interface. The timestamp returned by {@link PubSubMessage.getPubSubMessageTime()} refers to the time when Venice processed the event, and calls to this method will seek based on that sequence of events. Note: it bears no relation to timestamps provided by upstream producers when writing to Venice where a user may optionally provide a timestamp at time of producing a record.
      Specified by:
      seekToTimestamps in interface VeniceChangelogConsumer<K,V>
      Overrides:
      seekToTimestamps in class VeniceChangelogConsumerImpl<K,V>
      Parameters:
      timestamps - a map keyed by a partition ID, and the timestamp checkpoints to seek for each partition.
      Returns:

    • subscribe

      public CompletableFuture<Void> subscribe(Set<Integer> partitions)
      Description copied from interface: VeniceChangelogConsumer
      Subscribe a set of partitions for a store to this VeniceChangelogConsumer. The VeniceChangelogConsumer should try to consume messages from all partitions that are subscribed to it.
      Specified by:
      subscribe in interface VeniceChangelogConsumer<K,V>
      Overrides:
      subscribe in class VeniceChangelogConsumerImpl<K,V>
      Parameters:
      partitions - the set of partition to subscribe and consume
      Returns:
      a future which completes when the partitions are ready to be consumed data

    • seekToTail

      public CompletableFuture<Void> seekToTail(Set<Integer> partitions)
      Description copied from interface: VeniceChangelogConsumer
      Seek to the end of events which have been transmitted to Venice and start consuming new events. This will ONLY consume events transmitted via nearline and incremental push. It will not read batch push data.
      Specified by:
      seekToTail in interface VeniceChangelogConsumer<K,V>
      Overrides:
      seekToTail in class VeniceChangelogConsumerImpl<K,V>
      Parameters:
      partitions - the set of partitions to seek with
      Returns:
      a future which completes when the operation has succeeded for all partitions.

    • seekToEndOfPush

      public CompletableFuture<Void> seekToEndOfPush(Set<Integer> partitions)
      Description copied from interface: VeniceChangelogConsumer
      Seek to the end of the last push for a given set of partitions. This instructs the consumer to begin consuming events which are transmitted to Venice following the last batch push.
      Specified by:
      seekToEndOfPush in interface VeniceChangelogConsumer<K,V>
      Overrides:
      seekToEndOfPush in class VeniceChangelogConsumerImpl<K,V>
      Parameters:
      partitions - the set of partitions to seek with
      Returns:
      a future which completes when the operation has succeeded for all partitions.

    • subscribed

      public boolean subscribed()

    • internalSeek

      protected CompletableFuture<Void> internalSeek(Set<Integer> partitions, PubSubTopic targetTopic, com.linkedin.davinci.consumer.VeniceChangelogConsumerImpl.SeekFunction seekAction)
      Overrides:
      internalSeek in class VeniceChangelogConsumerImpl<K,V>