Package com.linkedin.davinci.consumer

Interface BootstrappingVeniceChangelogConsumer<K,V>

Type Parameters:
K -
V -
All Known Implementing Classes:
LocalBootstrappingVeniceChangelogConsumer
public interface BootstrappingVeniceChangelogConsumer<K,V>
This interface is meant for users where local state must be built off of the entirety of a venice data set (i.e. Non-idempotent event ingestion), rather than dealing with an event at a time. THIS IS EXPENSIVE. It's highly recommended that users use the VeniceChangelogConsumer interface as a means to consume Venice Change capture data. Implementations of this interface rely on access to a compacted view to the data and scanning the entirety of that compacted view initial calls to poll(). This is the only supported pattern with this interface today. VeniceChangelogConsumer enables finer control. This interface is intentionally limited as implementors rely on local checkpointing and maintenance of state which might be easily corrupted with byzantine seek() calls.

  • Method Details

    • start

      CompletableFuture<Void> start(Set<Integer> partitions)
      Start performs both a topic subscription and catch up. The client will look at the latest offset in the server and sync bootstrap data up to that point in changes. Once that is done for all partitions, the future will complete. NOTE: This future may take some time to complete depending on how much data needs to be ingested in order to catch up with the time that this client started.
      Parameters:
      partitions - which partition id's to catch up with
      Returns:
      a future that completes once catch up is complete for all passed in partitions.

    • start

      CompletableFuture<Void> start()

    • stop

      void stop() throws Exception
      Throws:
      Exception

    • poll

      Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>> poll(long timeoutInMs)
      polls for the next batch of change events. The first records returned following calling 'start()' will be from the bootstrap state. Once this state is consumed, subsequent calls to poll will be based off of recent updates to the Venice store.
      Parameters:
      timeoutInMs -
      Returns: