Package com.linkedin.davinci.consumer

Interface VeniceChangelogConsumer<K,​V>

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      void close()
      Release the internal resources.
      int getPartitionCount()  
      boolean isCaughtUp()
      Checks whether all subscribed partitions are caught up during bootstrap.
      void pause()
      Pause the client on all subscriptions.
      void pause​(java.util.Set<java.lang.Integer> partitions)
      Pause the client on all or subset of partitions this client is subscribed to.
      java.util.Collection<PubSubMessage<K,​ChangeEvent<V>,​VeniceChangeCoordinate>> poll​(long timeoutInMs)
      Polling function to get any available messages from the underlying system for all partitions subscribed.
      void resume()
      Pause the client on all subscriptions.
      void resume​(java.util.Set<java.lang.Integer> partitions)
      Resume the client on all or subset of partitions this client is subscribed to and has paused.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToBeginningOfPush()
      Seek to the beginning of the push for subscribed partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToBeginningOfPush​(java.util.Set<java.lang.Integer> partitions)
      Seek to the beginning of the push for a set of partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToCheckpoint​(java.util.Set<VeniceChangeCoordinate> checkpoints)
      Seek the provided checkpoints for the specified partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToEndOfPush()
      Seek to the end of the push for all subscribed partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToEndOfPush​(java.util.Set<java.lang.Integer> partitions)
      Seek to the end of the last push for a given set of partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToTail()
      Seek to the end of events which have been transmitted to Venice for all subscribed partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToTail​(java.util.Set<java.lang.Integer> partitions)
      Seek to the end of events which have been transmitted to Venice and start consuming new events.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToTimestamp​(java.lang.Long timestamp)
      Seek to the specified timestamp for all subscribed partitions.
      java.util.concurrent.CompletableFuture<java.lang.Void> seekToTimestamps​(java.util.Map<java.lang.Integer,​java.lang.Long> timestamps)
      Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture.
      java.util.concurrent.CompletableFuture<java.lang.Void> subscribe​(java.util.Set<java.lang.Integer> partitions)
      Subscribe a set of partitions for a store to this VeniceChangelogConsumer.
      java.util.concurrent.CompletableFuture<java.lang.Void> subscribeAll()
      Subscribe all partitions belonging to a specific store.
      void unsubscribe​(java.util.Set<java.lang.Integer> partitions)
      Stop ingesting messages from a set of partitions for a specific store.
      void unsubscribeAll()
      Stop ingesting messages from all partitions.

    • Method Detail

      • getPartitionCount

        int getPartitionCount()
        Returns:
        total number of store partitions

      • subscribe

        java.util.concurrent.CompletableFuture<java.lang.Void> subscribe​(java.util.Set<java.lang.Integer> partitions)
        Subscribe a set of partitions for a store to this VeniceChangelogConsumer. The VeniceChangelogConsumer should try to consume messages from all partitions that are subscribed to it.
        Parameters:
        partitions - the set of partition to subscribe and consume
        Returns:
        a future which completes when the partitions are ready to be consumed data
        Throws:
        a - VeniceException if subscribe operation failed for any of the partitions

      • seekToBeginningOfPush

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToBeginningOfPush​(java.util.Set<java.lang.Integer> partitions)
        Seek to the beginning of the push for a set of partitions. This is analogous to doing a bootstrap of data for the consumer. This seek will ONLY seek to the beginning of the version which is currently serving data, and the consumer will switch to reading data from a new version (should one get created) once it has read up to the point in the change capture stream that indicates the version swap (which can only occur after consuming all the data in the last push). This instructs the consumer to consume data from the batch push.
        Parameters:
        partitions - the set of partitions to seek with
        Returns:
        a future which completes when the operation has succeeded for all partitions.
        Throws:
        VeniceException - if seek operation failed for any of the partitions, or seeking was performed on unsubscribed partitions

      • seekToBeginningOfPush

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToBeginningOfPush()
        Seek to the beginning of the push for subscribed partitions. See seekToBeginningOfPush(Set) for more information.
        Returns:
        a future which completes when the partitions are ready to be consumed data
        Throws:
        VeniceException - if seek operation failed for any of the partitions.

      • seekToEndOfPush

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToEndOfPush​(java.util.Set<java.lang.Integer> partitions)
        Seek to the end of the last push for a given set of partitions. This instructs the consumer to begin consuming events which are transmitted to Venice following the last batch push.
        Parameters:
        partitions - the set of partitions to seek with
        Returns:
        a future which completes when the operation has succeeded for all partitions.
        Throws:
        VeniceException - if seek operation failed for any of the partitions, or seeking was performed on unsubscribed partitions

      • pause

        void pause()
        Pause the client on all subscriptions. See pause(Set) for more information.
        Throws:
        VeniceException - if operation failed for any of the partitions.

      • resume

        void resume​(java.util.Set<java.lang.Integer> partitions)
        Resume the client on all or subset of partitions this client is subscribed to and has paused.
        Throws:
        VeniceException - if operation failed for any of the partitions.

      • resume

        void resume()
        Pause the client on all subscriptions. See resume(Set) for more information.
        Throws:
        VeniceException - if operation failed for any of the partitions.

      • pause

        void pause​(java.util.Set<java.lang.Integer> partitions)
        Pause the client on all or subset of partitions this client is subscribed to. Calls to poll(long) will not return results from paused partitions until resume(Set) is called again later for those partitions.
        Throws:
        VeniceException - if operation failed for any of the partitions.

      • seekToEndOfPush

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToEndOfPush()
        Seek to the end of the push for all subscribed partitions. See seekToEndOfPush(Set) for more information.
        Returns:
        a future which completes when the operation has succeeded for all partitions.
        Throws:
        VeniceException - if seek operation failed for any of the partitions.

      • seekToTail

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToTail​(java.util.Set<java.lang.Integer> partitions)
        Seek to the end of events which have been transmitted to Venice and start consuming new events. This will ONLY consume events transmitted via nearline and incremental push. It will not read batch push data.
        Parameters:
        partitions - the set of partitions to seek with
        Returns:
        a future which completes when the operation has succeeded for all partitions.
        Throws:
        VeniceException - if seek operation failed for any of the partitions, or seeking was performed on unsubscribed partitions

      • seekToTail

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToTail()
        Seek to the end of events which have been transmitted to Venice for all subscribed partitions. See seekToTail(Set) for more information.
        Returns:
        a future which completes when the operation has succeeded for all partitions.
        Throws:
        VeniceException - if seek operation failed for any of the partitions.

      • seekToCheckpoint

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToCheckpoint​(java.util.Set<VeniceChangeCoordinate> checkpoints)
                                                                 throws VeniceCoordinateOutOfRangeException
        Seek the provided checkpoints for the specified partitions. Note about checkpoints: Checkpoints have the following properties and should be considered: -Checkpoints are NOT comparable or valid across partitions. -Checkpoints are NOT comparable or valid across colos -Checkpoints are NOT comparable across store versions -It is not possible to determine the number of events between two checkpoints -It is possible that a checkpoint is no longer on retention. In such case, we will return an exception to the caller.
        Parameters:
        checkpoints -
        Returns:
        a future which completes when seek has completed for all partitions
        Throws:
        VeniceException - if seek operation failed for any of the partitions
        VeniceCoordinateOutOfRangeException - if passed checkpoint is no longer valid

      • subscribeAll

        java.util.concurrent.CompletableFuture<java.lang.Void> subscribeAll()
        Subscribe all partitions belonging to a specific store.
        Returns:
        a future which completes when all partitions are ready to be consumed data
        Throws:
        a - VeniceException if subscribe operation failed for any of the partitions

      • seekToTimestamps

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToTimestamps​(java.util.Map<java.lang.Integer,​java.lang.Long> timestamps)
        Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture. Note, this API can only be used to seek on nearline data applied to the current serving version in Venice. This will not seek on data transmitted via Batch Push. If the provided timestamp is lower than the earliest timestamp on a given stream, the earliest event will be returned. THIS WILL NOT SEEK TO DATA WHICH WAS APPLIED ON A PREVIOUS VERSION. You should never seek back in time to a timestamp which is smaller than the current time - rewindTimeInSeconds configured in the hybrid settings for this Venice store. The timestamp passed to this function should be associated to timestamps processed by this interface. The timestamp returned by {@link PubSubMessage.getPubSubMessageTime()} refers to the time when Venice processed the event, and calls to this method will seek based on that sequence of events. Note: it bears no relation to timestamps provided by upstream producers when writing to Venice where a user may optionally provide a timestamp at time of producing a record.
        Parameters:
        timestamps - a map keyed by a partition ID, and the timestamp checkpoints to seek for each partition.
        Returns:
        Throws:
        VeniceException - if seek operations failed for any of the specified partitions.

      • seekToTimestamp

        java.util.concurrent.CompletableFuture<java.lang.Void> seekToTimestamp​(java.lang.Long timestamp)
        Seek to the specified timestamp for all subscribed partitions. See seekToTimestamps(Map) for more information.
        Returns:
        a future which completes when the operation has succeeded for all partitions.
        Throws:
        VeniceException - if seek operation failed for any of the partitions.

      • unsubscribe

        void unsubscribe​(java.util.Set<java.lang.Integer> partitions)
        Stop ingesting messages from a set of partitions for a specific store.
        Parameters:
        partitions - The set of topic partitions to unsubscribe
        Throws:
        a - VeniceException if unsubscribe operation failed for any of the partitions

      • unsubscribeAll

        void unsubscribeAll()
        Stop ingesting messages from all partitions.
        Throws:
        a - VeniceException if unsubscribe operation failed for any of the partitions

      • poll

        java.util.Collection<PubSubMessage<K,​ChangeEvent<V>,​VeniceChangeCoordinate>> poll​(long timeoutInMs)
        Polling function to get any available messages from the underlying system for all partitions subscribed.
        Parameters:
        timeoutInMs - The maximum time to block/wait in between two polling requests (must not be greater than Long.MAX_VALUE milliseconds)
        Returns:
        a collection of messages since the last fetch for the subscribed list of topic partitions
        Throws:
        a - VeniceException if polling operation fails

      • isCaughtUp

        boolean isCaughtUp()
        Checks whether all subscribed partitions are caught up during bootstrap. If a partition's (currentTimestamp - latestMessageTimestamp) is smaller or equal to 1 min, we consider this partition is caught up.
        Returns:
        True if all subscribed partitions have caught up.

      • close

        void close()
        Release the internal resources.