Package com.linkedin.davinci.kafka.consumer

Interface StoreIngestionService

All Superinterfaces:

IngestionMetadataRetriever

All Known Implementing Classes:

KafkaStoreIngestionService

public interface StoreIngestionService
extends IngestionMetadataRetriever

An interface for Store Ingestion Service for Venice.

Method Summary

Modifier and Type	Method	Description
void	addIngestionNotifier(VeniceNotifier notifier)	Adds Notifier to get Notifications for get various status of the consumption tasks like start, completed, progress and error states.
boolean	containsRunningConsumption(VeniceStoreVersionConfig veniceStore)	Check whether there is a running consumption task for given store.
boolean	containsRunningConsumption(java.lang.String topic)	Check whether there is a running consumption task for given store version topic.
void	demoteToStandby(VeniceStoreVersionConfig veniceStoreVersionConfig, int partitionId, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker checker)
AggVersionedIngestionStats	getAggVersionedIngestionStats()	Get AggVersionedStorageIngestionStats
java.util.Set<java.lang.String>	getIngestingTopicsWithVersionStatusNotOnline()	Get topic names that are currently maintained by the ingestion service with corresponding version status not in an online state.
StoreIngestionTask	getStoreIngestionTask(java.lang.String topic)
VeniceConfigLoader	getVeniceConfigLoader()
boolean	isPartitionConsuming(java.lang.String topic, int partitionId)	Check whether the specified partition is still being consumed
boolean	killConsumptionTask(java.lang.String topicName)	Kill all of running consumptions of given store.
void	promoteToLeader(VeniceStoreVersionConfig veniceStoreVersionConfig, int partitionId, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker checker)
void	recordIngestionFailure(java.lang.String storeName)
void	replaceAndAddTestNotifier(VeniceNotifier notifier)
default void	startConsumption(VeniceStoreVersionConfig veniceStore, int partitionId)
void	startConsumption(VeniceStoreVersionConfig veniceStore, int partitionId, java.util.Optional<LeaderFollowerStateType> leaderState)	Starts consuming messages from Kafka Partition corresponding to Venice Partition.
java.util.concurrent.CompletableFuture<java.lang.Void>	stopConsumption(VeniceStoreVersionConfig veniceStore, int partitionId)	Stops consuming messages from Kafka Partition corresponding to Venice Partition.
void	stopConsumptionAndWait(VeniceStoreVersionConfig veniceStore, int partitionId, int sleepSeconds, int numRetries, boolean whetherToResetOffset)	Stops consuming messages from Kafka Partition corresponding to Venice Partition and wait up to (sleepSeconds * numRetires) to make sure partition consumption is stopped.

Methods inherited from interface com.linkedin.davinci.storage.IngestionMetadataRetriever

getConsumptionSnapshots, getStoreVersionCompressionDictionary, getTopicPartitionIngestionContext

Method Detail

startConsumption

void startConsumption(VeniceStoreVersionConfig veniceStore,
                      int partitionId,
                      java.util.Optional<LeaderFollowerStateType> leaderState)

Starts consuming messages from Kafka Partition corresponding to Venice Partition.

Parameters:

veniceStore - Venice Store for the partition.

partitionId - Venice partition's id.

leaderState - Initial L/F state.

startConsumption

default void startConsumption(VeniceStoreVersionConfig veniceStore,
                              int partitionId)

stopConsumption

java.util.concurrent.CompletableFuture<java.lang.Void> stopConsumption(VeniceStoreVersionConfig veniceStore,
                                                                       int partitionId)

Stops consuming messages from Kafka Partition corresponding to Venice Partition.

Parameters:

veniceStore - Venice Store for the partition.

partitionId - Venice partition's id.

stopConsumptionAndWait

void stopConsumptionAndWait(VeniceStoreVersionConfig veniceStore,
                            int partitionId,
                            int sleepSeconds,
                            int numRetries,
                            boolean whetherToResetOffset)

Stops consuming messages from Kafka Partition corresponding to Venice Partition and wait up to (sleepSeconds * numRetires) to make sure partition consumption is stopped.

killConsumptionTask

boolean killConsumptionTask(java.lang.String topicName)

Kill all of running consumptions of given store.

Parameters:

topicName - Venice topic (store and version number) for the corresponding consumer task that needs to be killed.

promoteToLeader

void promoteToLeader(VeniceStoreVersionConfig veniceStoreVersionConfig,
                     int partitionId,
                     LeaderFollowerPartitionStateModel.LeaderSessionIdChecker checker)

demoteToStandby

void demoteToStandby(VeniceStoreVersionConfig veniceStoreVersionConfig,
                     int partitionId,
                     LeaderFollowerPartitionStateModel.LeaderSessionIdChecker checker)

addIngestionNotifier

void addIngestionNotifier(VeniceNotifier notifier)

Adds Notifier to get Notifications for get various status of the consumption tasks like start, completed, progress and error states. Multiple Notifiers can be added for the same consumption tasks and all of them will be notified in order.

Parameters:

notifier -

replaceAndAddTestNotifier

void replaceAndAddTestNotifier(VeniceNotifier notifier)

containsRunningConsumption

boolean containsRunningConsumption(VeniceStoreVersionConfig veniceStore)

Check whether there is a running consumption task for given store.

containsRunningConsumption

boolean containsRunningConsumption(java.lang.String topic)

Check whether there is a running consumption task for given store version topic.

isPartitionConsuming

boolean isPartitionConsuming(java.lang.String topic,
                             int partitionId)

Check whether the specified partition is still being consumed

getIngestingTopicsWithVersionStatusNotOnline

java.util.Set<java.lang.String> getIngestingTopicsWithVersionStatusNotOnline()

Get topic names that are currently maintained by the ingestion service with corresponding version status not in an online state. Topics with invalid store or version number are also included in the returned list.

Returns:

a Set of topic names.

recordIngestionFailure

void recordIngestionFailure(java.lang.String storeName)

getAggVersionedIngestionStats

AggVersionedIngestionStats getAggVersionedIngestionStats()

Get AggVersionedStorageIngestionStats

Returns:

an instance of AggVersionedIngestionStats

getStoreIngestionTask

StoreIngestionTask getStoreIngestionTask(java.lang.String topic)

getVeniceConfigLoader

VeniceConfigLoader getVeniceConfigLoader()