Package com.linkedin.davinci.ingestion
Class IsolatedIngestionBackend
- java.lang.Object
-
- com.linkedin.davinci.ingestion.DefaultIngestionBackend
-
- com.linkedin.davinci.ingestion.IsolatedIngestionBackend
-
- All Implemented Interfaces:
DaVinciIngestionBackend
,IngestionBackendBase
,VeniceIngestionBackend
,java.io.Closeable
,java.lang.AutoCloseable
public class IsolatedIngestionBackend extends DefaultIngestionBackend implements DaVinciIngestionBackend, VeniceIngestionBackend
This class is the implementation of ingestion backend designed for ingestion isolation. It contains references to local ingestion components - including storage metadata service, storage service and store ingestion service that serves local ingestion, as well as ingestion request client that sends commands to isolated ingestion service process and ingestion listener that listens to ingestion reports from child process. Since RocksDB storage can only be owned by a single process, we have decided to keep metadata partition storage opened in child process and in the main process, we rely onMainIngestionStorageMetadataService
to serve as the in-memory metadata cache and persist the metadata updates from main process to metadata partition in child process. Topic partition ingestion requests will first be sent to child process and after COMPLETED is reported, they will be re-subscribed in main process to serve read traffics for user application and receive future updates. The implementation of APIs in this class should consider the states in both main process and child process, as we need to make sure we send the command to the correct process which holds the target storage engine.
-
-
Constructor Summary
Constructors Constructor Description IsolatedIngestionBackend(VeniceConfigLoader configLoader, ReadOnlyStoreRepository storeRepository, io.tehuti.metrics.MetricsRepository metricsRepository, StorageMetadataService storageMetadataService, KafkaStoreIngestionService storeIngestionService, StorageService storageService)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
addIngestionNotifier(VeniceNotifier ingestionListener)
void
addPushStatusNotifier(VeniceNotifier pushStatusNotifier)
Adds ingestion listener which reports ingestion status for different push monitors.void
close()
void
demoteToStandby(VeniceStoreVersionConfig storeConfig, int partition, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)
void
dropStoragePartitionGracefully(VeniceStoreVersionConfig storeConfig, int partition, int timeoutInSeconds, boolean removeEmptyStorageEngine)
This method stops to subscribe the specified topic partition and delete partition data from storage.java.lang.Process
getIsolatedIngestionServiceProcess()
java.util.Map<java.lang.String,java.util.Set<java.lang.Integer>>
getLoadedStoreUserPartitionsMapping()
Get currently loaded store partition mappings.MainIngestionMonitorService
getMainIngestionMonitorService()
MainIngestionRequestClient
getMainIngestionRequestClient()
boolean
isShuttingDown()
void
killConsumptionTask(java.lang.String topicName)
void
prepareForShutdown()
void
promoteToLeader(VeniceStoreVersionConfig storeConfig, int partition, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)
void
removeStorageEngine(java.lang.String topicName)
void
setIsolatedIngestionServiceProcess(java.lang.Process process)
void
shutdownIngestionTask(java.lang.String topicName)
void
startConsumption(VeniceStoreVersionConfig storeConfig, int partition, java.util.Optional<LeaderFollowerStateType> leaderState)
java.util.concurrent.CompletableFuture<java.lang.Void>
stopConsumption(VeniceStoreVersionConfig storeConfig, int partition)
-
Methods inherited from class com.linkedin.davinci.ingestion.DefaultIngestionBackend
getStoreIngestionService, replaceAndAddTestPushStatusNotifier, setStorageEngineReference
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.linkedin.davinci.ingestion.DaVinciIngestionBackend
setStorageEngineReference
-
Methods inherited from interface com.linkedin.davinci.ingestion.IngestionBackendBase
dropStoragePartitionGracefully, getStoreIngestionService, startConsumption
-
Methods inherited from interface com.linkedin.davinci.ingestion.VeniceIngestionBackend
replaceAndAddTestPushStatusNotifier
-
-
-
-
Constructor Detail
-
IsolatedIngestionBackend
public IsolatedIngestionBackend(VeniceConfigLoader configLoader, ReadOnlyStoreRepository storeRepository, io.tehuti.metrics.MetricsRepository metricsRepository, StorageMetadataService storageMetadataService, KafkaStoreIngestionService storeIngestionService, StorageService storageService)
-
-
Method Detail
-
startConsumption
public void startConsumption(VeniceStoreVersionConfig storeConfig, int partition, java.util.Optional<LeaderFollowerStateType> leaderState)
- Specified by:
startConsumption
in interfaceIngestionBackendBase
- Overrides:
startConsumption
in classDefaultIngestionBackend
-
stopConsumption
public java.util.concurrent.CompletableFuture<java.lang.Void> stopConsumption(VeniceStoreVersionConfig storeConfig, int partition)
- Specified by:
stopConsumption
in interfaceIngestionBackendBase
- Overrides:
stopConsumption
in classDefaultIngestionBackend
-
promoteToLeader
public void promoteToLeader(VeniceStoreVersionConfig storeConfig, int partition, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)
- Specified by:
promoteToLeader
in interfaceVeniceIngestionBackend
- Overrides:
promoteToLeader
in classDefaultIngestionBackend
-
demoteToStandby
public void demoteToStandby(VeniceStoreVersionConfig storeConfig, int partition, LeaderFollowerPartitionStateModel.LeaderSessionIdChecker leaderSessionIdChecker)
- Specified by:
demoteToStandby
in interfaceVeniceIngestionBackend
- Overrides:
demoteToStandby
in classDefaultIngestionBackend
-
dropStoragePartitionGracefully
public void dropStoragePartitionGracefully(VeniceStoreVersionConfig storeConfig, int partition, int timeoutInSeconds, boolean removeEmptyStorageEngine)
Description copied from interface:IngestionBackendBase
This method stops to subscribe the specified topic partition and delete partition data from storage.- Specified by:
dropStoragePartitionGracefully
in interfaceIngestionBackendBase
- Overrides:
dropStoragePartitionGracefully
in classDefaultIngestionBackend
- Parameters:
storeConfig
- Store version configpartition
- Partition number to be dropped in the store version.timeoutInSeconds
- Number of seconds to wait before timeout.removeEmptyStorageEngine
- Whether to drop storage engine when dropping the last partition.
-
removeStorageEngine
public void removeStorageEngine(java.lang.String topicName)
- Specified by:
removeStorageEngine
in interfaceDaVinciIngestionBackend
- Overrides:
removeStorageEngine
in classDefaultIngestionBackend
-
killConsumptionTask
public void killConsumptionTask(java.lang.String topicName)
- Specified by:
killConsumptionTask
in interfaceIngestionBackendBase
- Overrides:
killConsumptionTask
in classDefaultIngestionBackend
-
shutdownIngestionTask
public void shutdownIngestionTask(java.lang.String topicName)
- Specified by:
shutdownIngestionTask
in interfaceIngestionBackendBase
- Overrides:
shutdownIngestionTask
in classDefaultIngestionBackend
-
addIngestionNotifier
public void addIngestionNotifier(VeniceNotifier ingestionListener)
- Specified by:
addIngestionNotifier
in interfaceIngestionBackendBase
- Overrides:
addIngestionNotifier
in classDefaultIngestionBackend
-
addPushStatusNotifier
public void addPushStatusNotifier(VeniceNotifier pushStatusNotifier)
Description copied from interface:VeniceIngestionBackend
Adds ingestion listener which reports ingestion status for different push monitors.- Specified by:
addPushStatusNotifier
in interfaceVeniceIngestionBackend
- Overrides:
addPushStatusNotifier
in classDefaultIngestionBackend
-
prepareForShutdown
public void prepareForShutdown()
- Specified by:
prepareForShutdown
in interfaceVeniceIngestionBackend
-
getLoadedStoreUserPartitionsMapping
public java.util.Map<java.lang.String,java.util.Set<java.lang.Integer>> getLoadedStoreUserPartitionsMapping()
Description copied from interface:IngestionBackendBase
Get currently loaded store partition mappings.- Specified by:
getLoadedStoreUserPartitionsMapping
in interfaceIngestionBackendBase
- Overrides:
getLoadedStoreUserPartitionsMapping
in classDefaultIngestionBackend
-
setIsolatedIngestionServiceProcess
public void setIsolatedIngestionServiceProcess(java.lang.Process process)
-
getIsolatedIngestionServiceProcess
public java.lang.Process getIsolatedIngestionServiceProcess()
-
getMainIngestionMonitorService
public MainIngestionMonitorService getMainIngestionMonitorService()
-
getMainIngestionRequestClient
public MainIngestionRequestClient getMainIngestionRequestClient()
-
isShuttingDown
public boolean isShuttingDown()
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Overrides:
close
in classDefaultIngestionBackend
-
-