Interface BootstrappingVeniceChangelogConsumer<K,​V>

  • Type Parameters:
    K -
    V -
    All Known Implementing Classes:
    LocalBootstrappingVeniceChangelogConsumer

    public interface BootstrappingVeniceChangelogConsumer<K,​V>
    This interface is meant for users where local state must be built off of the entirety of a venice data set (i.e. Non-idempotent event ingestion), rather than dealing with an event at a time. THIS IS EXPENSIVE. It's highly recommended that users use the VeniceChangelogConsumer interface as a means to consume Venice Change capture data. Implementations of this interface rely on access to a compacted view to the data and scanning the entirety of that compacted view initial calls to poll(). This is the only supported pattern with this interface today. VeniceChangelogConsumer enables finer control. This interface is intentionally limited as implementors rely on local checkpointing and maintenance of state which might be easily corrupted with byzantine seek() calls.
    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      java.util.Collection<PubSubMessage<K,​ChangeEvent<V>,​VeniceChangeCoordinate>> poll​(long timeoutInMs)
      polls for the next batch of change events.
      java.util.concurrent.CompletableFuture<java.lang.Void> start()  
      java.util.concurrent.CompletableFuture<java.lang.Void> start​(java.util.Set<java.lang.Integer> partitions)
      Start performs both a topic subscription and catch up.
      void stop()  
    • Method Detail

      • start

        java.util.concurrent.CompletableFuture<java.lang.Void> start​(java.util.Set<java.lang.Integer> partitions)
        Start performs both a topic subscription and catch up. The client will look at the latest offset in the server and sync bootstrap data up to that point in changes. Once that is done for all partitions, the future will complete. NOTE: This future may take some time to complete depending on how much data needs to be ingested in order to catch up with the time that this client started.
        Parameters:
        partitions - which partition id's to catch up with
        Returns:
        a future that completes once catch up is complete for all passed in partitions.
      • start

        java.util.concurrent.CompletableFuture<java.lang.Void> start()
      • stop

        void stop()
           throws java.lang.Exception
        Throws:
        java.lang.Exception
      • poll

        java.util.Collection<PubSubMessage<K,​ChangeEvent<V>,​VeniceChangeCoordinate>> poll​(long timeoutInMs)
        polls for the next batch of change events. The first records returned following calling 'start()' will be from the bootstrap state. Once this state is consumed, subsequent calls to poll will be based off of recent updates to the Venice store.
        Parameters:
        timeoutInMs -
        Returns: