Package com.linkedin.davinci.ingestion

Class DefaultIngestionBackend

java.lang.Object

com.linkedin.davinci.ingestion.DefaultIngestionBackend

All Implemented Interfaces:

DaVinciIngestionBackend, IngestionBackendBase, VeniceIngestionBackend, java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

IsolatedIngestionBackend

public class DefaultIngestionBackend
extends java.lang.Object
implements DaVinciIngestionBackend, VeniceIngestionBackend

The default ingestion backend implementation. Ingestion will be done in the same JVM as the application.

Constructor Summary

Constructors

Constructor	Description
DefaultIngestionBackend(StorageMetadataService storageMetadataService, KafkaStoreIngestionService storeIngestionService, StorageService storageService)

Method Summary

Modifier and Type	Method	Description
void	addIngestionNotifier(VeniceNotifier ingestionListener)
void	addPushStatusNotifier(VeniceNotifier pushStatusNotifier)	Adds ingestion listener which reports ingestion status for different push monitors.
void	close()
void	demoteToStandby(VeniceStoreVersionConfig storeConfig, int partition, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)
void	dropStoragePartitionGracefully(VeniceStoreVersionConfig storeConfig, int partition, int timeoutInSeconds, boolean removeEmptyStorageEngine)	This method stops to subscribe the specified topic partition and delete partition data from storage.
java.util.Map<java.lang.String,java.util.Set<java.lang.Integer>>	getLoadedStoreUserPartitionsMapping()	Get currently loaded store partition mappings.
KafkaStoreIngestionService	getStoreIngestionService()
void	killConsumptionTask(java.lang.String topicName)
void	promoteToLeader(VeniceStoreVersionConfig storeConfig, int partition, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)
void	removeStorageEngine(java.lang.String topicName)
void	replaceAndAddTestPushStatusNotifier(VeniceNotifier pushStatusNotifier)
void	setStorageEngineReference(java.lang.String topicName, java.util.concurrent.atomic.AtomicReference<AbstractStorageEngine> storageEngineReference)
void	shutdownIngestionTask(java.lang.String topicName)
void	startConsumption(VeniceStoreVersionConfig storeConfig, int partition, java.util.Optional<LeaderFollowerStateType> leaderState)
java.util.concurrent.CompletableFuture<java.lang.Void>	stopConsumption(VeniceStoreVersionConfig storeConfig, int partition)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.davinci.ingestion.IngestionBackendBase

dropStoragePartitionGracefully, startConsumption

Methods inherited from interface com.linkedin.davinci.ingestion.VeniceIngestionBackend

prepareForShutdown

Constructor Detail

DefaultIngestionBackend

public DefaultIngestionBackend(StorageMetadataService storageMetadataService,
                               KafkaStoreIngestionService storeIngestionService,
                               StorageService storageService)

Method Detail

startConsumption

public void startConsumption(VeniceStoreVersionConfig storeConfig,
                             int partition,
                             java.util.Optional<LeaderFollowerStateType> leaderState)

Specified by:

startConsumption in interface IngestionBackendBase

stopConsumption

public java.util.concurrent.CompletableFuture<java.lang.Void> stopConsumption(VeniceStoreVersionConfig storeConfig,
                                                                              int partition)

Specified by:

stopConsumption in interface IngestionBackendBase

killConsumptionTask

public void killConsumptionTask(java.lang.String topicName)

Specified by:

killConsumptionTask in interface IngestionBackendBase

shutdownIngestionTask

public void shutdownIngestionTask(java.lang.String topicName)

Specified by:

shutdownIngestionTask in interface IngestionBackendBase

removeStorageEngine

public void removeStorageEngine(java.lang.String topicName)

Specified by:

removeStorageEngine in interface DaVinciIngestionBackend

dropStoragePartitionGracefully

public void dropStoragePartitionGracefully(VeniceStoreVersionConfig storeConfig,
                                           int partition,
                                           int timeoutInSeconds,
                                           boolean removeEmptyStorageEngine)

Description copied from interface: IngestionBackendBase

This method stops to subscribe the specified topic partition and delete partition data from storage.

Specified by:

dropStoragePartitionGracefully in interface IngestionBackendBase

Parameters:

storeConfig - Store version config

partition - Partition number to be dropped in the store version.

timeoutInSeconds - Number of seconds to wait before timeout.

removeEmptyStorageEngine - Whether to drop storage engine when dropping the last partition.

promoteToLeader

public void promoteToLeader(VeniceStoreVersionConfig storeConfig,
                            int partition,
                            LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)

Specified by:

promoteToLeader in interface VeniceIngestionBackend

demoteToStandby

public void demoteToStandby(VeniceStoreVersionConfig storeConfig,
                            int partition,
                            LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)

Specified by:

demoteToStandby in interface VeniceIngestionBackend

addIngestionNotifier

public void addIngestionNotifier(VeniceNotifier ingestionListener)

Specified by:

addIngestionNotifier in interface IngestionBackendBase

addPushStatusNotifier

public void addPushStatusNotifier(VeniceNotifier pushStatusNotifier)

Description copied from interface: VeniceIngestionBackend

Adds ingestion listener which reports ingestion status for different push monitors.

Specified by:

addPushStatusNotifier in interface VeniceIngestionBackend

replaceAndAddTestPushStatusNotifier

public void replaceAndAddTestPushStatusNotifier(VeniceNotifier pushStatusNotifier)

Specified by:

replaceAndAddTestPushStatusNotifier in interface VeniceIngestionBackend

setStorageEngineReference

public void setStorageEngineReference(java.lang.String topicName,
                                      java.util.concurrent.atomic.AtomicReference<AbstractStorageEngine> storageEngineReference)

Specified by:

setStorageEngineReference in interface DaVinciIngestionBackend

getStoreIngestionService

public KafkaStoreIngestionService getStoreIngestionService()

Specified by:

getStoreIngestionService in interface IngestionBackendBase

getLoadedStoreUserPartitionsMapping

public java.util.Map<java.lang.String,java.util.Set<java.lang.Integer>> getLoadedStoreUserPartitionsMapping()

Description copied from interface: IngestionBackendBase

Get currently loaded store partition mappings.

Specified by:

getLoadedStoreUserPartitionsMapping in interface IngestionBackendBase

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable