Package com.linkedin.davinci.consumer

Class BootstrappingVeniceChangelogConsumerDaVinciRecordTransformerImpl<K,V>

java.lang.Object
com.linkedin.davinci.consumer.BootstrappingVeniceChangelogConsumerDaVinciRecordTransformerImpl<K,V>
All Implemented Interfaces:
BootstrappingVeniceChangelogConsumer<K,V>
@Experimental public class BootstrappingVeniceChangelogConsumerDaVinciRecordTransformerImpl<K,V> extends Object implements BootstrappingVeniceChangelogConsumer<K,V>

  • Constructor Details

    • BootstrappingVeniceChangelogConsumerDaVinciRecordTransformerImpl

      public BootstrappingVeniceChangelogConsumerDaVinciRecordTransformerImpl(ChangelogClientConfig changelogClientConfig)

  • Method Details

    • start

      public CompletableFuture<Void> start(Set<Integer> partitions)
      Description copied from interface: BootstrappingVeniceChangelogConsumer
      Start performs both a topic subscription and catch up. The client will look at the latest offset in the server and sync bootstrap data up to that point in changes. Once that is done for all partitions, the future will complete. NOTE: This future may take some time to complete depending on how much data needs to be ingested in order to catch up with the time that this client started. NOTE: In the experimental client, the future will complete when there is at least one message to be polled. We don't wait for all partitions to catch up, as loading every message into a buffer will result in an Out Of Memory error. Instead, use the BootstrappingVeniceChangelogConsumer.isCaughtUp() method to determine once all subscribed partitions have caught up. NOTE: In the experimental client, if you pass in an empty set, it will subscribe to all partitions for the store
      Specified by:
      start in interface BootstrappingVeniceChangelogConsumer<K,V>
      Parameters:
      partitions - which partition id's to catch up with
      Returns:
      a future that completes once catch up is complete for all passed in partitions.

    • start

      public CompletableFuture<Void> start()
      Specified by:
      start in interface BootstrappingVeniceChangelogConsumer<K,V>

    • stop

      public void stop() throws Exception
      Specified by:
      stop in interface BootstrappingVeniceChangelogConsumer<K,V>
      Throws:
      Exception

    • poll

      public Collection<PubSubMessage<K,ChangeEvent<V>,VeniceChangeCoordinate>> poll(long timeoutInMs)
      Description copied from interface: BootstrappingVeniceChangelogConsumer
      polls for the next batch of change events. The first records returned following calling 'start()' will be from the bootstrap state. Once this state is consumed, subsequent calls to poll will be based off of recent updates to the Venice store. In the experimental client, records will be returned in batches configured to the MAX_BUFFER_SIZE. So the initial calls to poll will be from records from the bootstrap state, until the partitions have caught up. Additionally, if the buffer hits the MAX_BUFFER_SIZE before the timeout is hit, poll will return immediately.
      Specified by:
      poll in interface BootstrappingVeniceChangelogConsumer<K,V>
      Returns:

    • isCaughtUp

      public boolean isCaughtUp()
      Description copied from interface: BootstrappingVeniceChangelogConsumer
      In the experimental client, once this becomes true it will stay true even if we start to lag after the bootstrapping phase.
      Specified by:
      isCaughtUp in interface BootstrappingVeniceChangelogConsumer<K,V>
      Returns:
      True if all subscribed partitions have caught up.

    • getRecordTransformerConfig

      public DaVinciRecordTransformerConfig getRecordTransformerConfig()

    • setBackgroundReporterThreadSleepIntervalSeconds

      protected void setBackgroundReporterThreadSleepIntervalSeconds(long interval)