Package com.linkedin.davinci.kafka.consumer

Interface StoreIngestionService

All Known Implementing Classes:
KafkaStoreIngestionService
public interface StoreIngestionService
An interface for Store Ingestion Service for Venice.

  • Method Details

    • startConsumption

      void startConsumption(VeniceStoreVersionConfig veniceStore, int partitionId, Optional<PubSubPosition> pubSubPosition)
      Starts consuming messages from Kafka Partition corresponding to Venice Partition.
      Parameters:
      veniceStore - Venice Store for the partition.
      partitionId - Venice partition's id.
      pubSubPosition -

    • stopConsumption

      CompletableFuture<Void> stopConsumption(VeniceStoreVersionConfig veniceStore, int partitionId)
      Stops consuming messages from Kafka Partition corresponding to Venice Partition.
      Parameters:
      veniceStore - Venice Store for the partition.
      partitionId - Venice partition's id.

    • stopConsumptionAndWait

      void stopConsumptionAndWait(VeniceStoreVersionConfig veniceStore, int partitionId, int sleepSeconds, int numRetries, boolean whetherToResetOffset)
      Stops consuming messages from Kafka Partition corresponding to Venice Partition and wait up to (sleepSeconds * numRetires) to make sure partition consumption is stopped.

    • killConsumptionTask

      boolean killConsumptionTask(String topicName)
      Kill all of running consumptions of given store.
      Parameters:
      topicName - Venice topic (store and version number) for the corresponding consumer task that needs to be killed.

    • promoteToLeader

      void promoteToLeader(VeniceStoreVersionConfig veniceStoreVersionConfig, int partitionId, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker checker)

    • demoteToStandby

      void demoteToStandby(VeniceStoreVersionConfig veniceStoreVersionConfig, int partitionId, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker checker)

    • addIngestionNotifier

      void addIngestionNotifier(VeniceNotifier notifier)
      Adds Notifier to get Notifications for get various status of the consumption tasks like start, completed, progress and error states. Multiple Notifiers can be added for the same consumption tasks and all of them will be notified in order.
      Parameters:
      notifier -

    • replaceAndAddTestNotifier

      void replaceAndAddTestNotifier(VeniceNotifier notifier)

    • containsRunningConsumption

      boolean containsRunningConsumption(VeniceStoreVersionConfig veniceStore)
      Check whether there is a running consumption task for given store.

    • containsRunningConsumption

      boolean containsRunningConsumption(String topic)
      Check whether there is a running consumption task for given store version topic.

    • isPartitionConsuming

      boolean isPartitionConsuming(String topic, int partitionId)
      Check whether the specified partition is still being consumed

    • getIngestingTopicsWithVersionStatusNotOnline

      Set<String> getIngestingTopicsWithVersionStatusNotOnline()
      Get topic names that are currently maintained by the ingestion service with corresponding version status not in an online state. Topics with invalid store or version number are also included in the returned list.
      Returns:
      a Set of topic names.

    • recordIngestionFailure

      void recordIngestionFailure(String storeName)

    • getAggVersionedIngestionStats

      AggVersionedIngestionStats getAggVersionedIngestionStats()
      Get AggVersionedStorageIngestionStats
      Returns:
      an instance of AggVersionedIngestionStats

    • getStoreIngestionTask

      StoreIngestionTask getStoreIngestionTask(String topic)

    • getVeniceConfigLoader

      VeniceConfigLoader getVeniceConfigLoader()

    • getVeniceWriterFactory

      VeniceWriterFactory getVeniceWriterFactory()

    • getPubSubPosition

      Optional<PubSubPosition> getPubSubPosition(VeniceStoreVersionConfig veniceStore, int partitionId, Long timestamp, PubSubPosition pubSubPosition)