Package com.linkedin.davinci.storage

Class StorageService

java.lang.Object

com.linkedin.venice.service.AbstractVeniceService

com.linkedin.davinci.storage.StorageService

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class StorageService
extends AbstractVeniceService

Storage interface to Venice Server, Da Vinci and Isolated Ingestion Service. Manages creation and deletion of storage engines and partitions. Use StorageEngineRepository, if read only access is desired for the Storage Engines.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.venice.service.AbstractVeniceService

AbstractVeniceService.ServiceState

Field Summary

Fields inherited from class com.linkedin.venice.service.AbstractVeniceService

logger, serviceState

Constructor Summary

Constructors

Constructor	Description
StorageService(VeniceConfigLoader configLoader, AggVersionedStorageEngineStats storageEngineStats, RocksDBMemoryStats rocksDBMemoryStats, InternalAvroSpecificSerializer<StoreVersionState> storeVersionStateSerializer, InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer, ReadOnlyStoreRepository storeRepository)
StorageService(VeniceConfigLoader configLoader, AggVersionedStorageEngineStats storageEngineStats, RocksDBMemoryStats rocksDBMemoryStats, InternalAvroSpecificSerializer<StoreVersionState> storeVersionStateSerializer, InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer, ReadOnlyStoreRepository storeRepository, boolean restoreDataPartitions, boolean restoreMetadataPartitions)
StorageService(VeniceConfigLoader configLoader, AggVersionedStorageEngineStats storageEngineStats, RocksDBMemoryStats rocksDBMemoryStats, InternalAvroSpecificSerializer<StoreVersionState> storeVersionStateSerializer, InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer, ReadOnlyStoreRepository storeRepository, boolean restoreDataPartitions, boolean restoreMetadataPartitions, java.util.function.Function<java.lang.String,java.lang.Boolean> checkWhetherStorageEngineShouldBeKeptOrNot)

Method Summary

Modifier and Type	Method	Description
void	checkWhetherStoragePartitionsShouldBeKeptOrNot(SafeHelixManager manager)
void	cleanupAllStores(VeniceConfigLoader configLoader)
void	closeAllStorageEngines()
void	closeStorageEngine(java.lang.String kafkaTopic)
void	closeStorePartition(VeniceStoreVersionConfig storeConfig, int partition)
void	dropStorePartition(VeniceStoreVersionConfig storeConfig, int partition)	Drops the partition of the specified store version in the storage service.
void	dropStorePartition(VeniceStoreVersionConfig storeConfig, int partition, boolean removeEmptyStorageEngine)	Drops the partition of the specified store version in the storage service.
void	forceStorageEngineCleanup(java.lang.String kafkaTopic)	This function is used to forcely clean up all the databases belonging to {@param kafkaTopic}.
StorageEngineFactory	getInternalStorageEngineFactory(VeniceStoreVersionConfig storeConfig)	This method should ideally be Private, but marked as public for validating the result.
java.util.Optional<org.rocksdb.Statistics>	getRocksDBAggregatedStatistics()
AbstractStorageEngine	getStorageEngine(java.lang.String kafkaTopic)
StorageEngineRepository	getStorageEngineRepository()
java.util.Map<java.lang.String,java.util.Set<java.lang.Integer>>	getStoreAndUserPartitionsMapping()
java.util.function.BiConsumer<java.lang.String,StoreVersionState>	getStoreVersionStateSyncer()
java.util.List<java.lang.Integer>	getUserPartitions(java.lang.String kafkaTopicName)
AbstractStorageEngine	openStore(VeniceStoreVersionConfig storeConfig, java.util.function.Supplier<StoreVersionState> initialStoreVersionStateSupplier)	Creates a StorageEngineFactory for the persistence type if not already present.
AbstractStorageEngine	openStoreForNewPartition(VeniceStoreVersionConfig storeConfig, int partitionId, java.util.function.Supplier<StoreVersionState> initialStoreVersionStateSupplier)
void	removeStorageEngine(java.lang.String kafkaTopic)
boolean	startInner()
void	stopInner()

Methods inherited from class com.linkedin.venice.service.AbstractVeniceService

close, getName, isRunning, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageService

public StorageService(VeniceConfigLoader configLoader,
                      AggVersionedStorageEngineStats storageEngineStats,
                      RocksDBMemoryStats rocksDBMemoryStats,
                      InternalAvroSpecificSerializer<StoreVersionState> storeVersionStateSerializer,
                      InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer,
                      ReadOnlyStoreRepository storeRepository,
                      boolean restoreDataPartitions,
                      boolean restoreMetadataPartitions,
                      java.util.function.Function<java.lang.String,java.lang.Boolean> checkWhetherStorageEngineShouldBeKeptOrNot)

StorageService

public StorageService(VeniceConfigLoader configLoader,
                      AggVersionedStorageEngineStats storageEngineStats,
                      RocksDBMemoryStats rocksDBMemoryStats,
                      InternalAvroSpecificSerializer<StoreVersionState> storeVersionStateSerializer,
                      InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer,
                      ReadOnlyStoreRepository storeRepository,
                      boolean restoreDataPartitions,
                      boolean restoreMetadataPartitions)

StorageService

public StorageService(VeniceConfigLoader configLoader,
                      AggVersionedStorageEngineStats storageEngineStats,
                      RocksDBMemoryStats rocksDBMemoryStats,
                      InternalAvroSpecificSerializer<StoreVersionState> storeVersionStateSerializer,
                      InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer,
                      ReadOnlyStoreRepository storeRepository)

See Also:

StorageService(VeniceConfigLoader, AggVersionedStorageEngineStats, RocksDBMemoryStats, InternalAvroSpecificSerializer, InternalAvroSpecificSerializer, ReadOnlyStoreRepository)

Method Detail

openStoreForNewPartition

public AbstractStorageEngine openStoreForNewPartition(VeniceStoreVersionConfig storeConfig,
                                                      int partitionId,
                                                      java.util.function.Supplier<StoreVersionState> initialStoreVersionStateSupplier)

getStoreVersionStateSyncer

public java.util.function.BiConsumer<java.lang.String,StoreVersionState> getStoreVersionStateSyncer()

getInternalStorageEngineFactory

public StorageEngineFactory getInternalStorageEngineFactory(VeniceStoreVersionConfig storeConfig)

This method should ideally be Private, but marked as public for validating the result.

Parameters:

storeConfig - StoreConfig of the store.

Returns:

Factory corresponding to the store.

getRocksDBAggregatedStatistics

public java.util.Optional<org.rocksdb.Statistics> getRocksDBAggregatedStatistics()

openStore

public AbstractStorageEngine openStore(VeniceStoreVersionConfig storeConfig,
                                       java.util.function.Supplier<StoreVersionState> initialStoreVersionStateSupplier)

Creates a StorageEngineFactory for the persistence type if not already present. Creates a new storage engine for the given store in the factory and registers the storage engine with the store repository.

Parameters:

storeConfig - The store specific properties

initialStoreVersionStateSupplier - invoked to initialize the SVS when a brand-new storage engine is created

Returns:

StorageEngine that was created for the given store definition.

checkWhetherStoragePartitionsShouldBeKeptOrNot

public void checkWhetherStoragePartitionsShouldBeKeptOrNot(SafeHelixManager manager)

dropStorePartition

public void dropStorePartition(VeniceStoreVersionConfig storeConfig,
                               int partition)

Drops the partition of the specified store version in the storage service. When all data partitions are dropped, it will also drop the storage engine of the specific store version.

Parameters:

storeConfig - config of the store version.

partition - partition ID to be dropped.

dropStorePartition

public void dropStorePartition(VeniceStoreVersionConfig storeConfig,
                               int partition,
                               boolean removeEmptyStorageEngine)

Drops the partition of the specified store version in the storage service.

Parameters:

storeConfig - config of the store version.

partition - partition ID to be dropped.

removeEmptyStorageEngine - Whether to delete the storage engine when there is no remaining data partition.

closeStorePartition

public void closeStorePartition(VeniceStoreVersionConfig storeConfig,
                                int partition)

removeStorageEngine

public void removeStorageEngine(java.lang.String kafkaTopic)

forceStorageEngineCleanup

public void forceStorageEngineCleanup(java.lang.String kafkaTopic)

This function is used to forcely clean up all the databases belonging to {@param kafkaTopic}. This function will only be used when the removeStorageEngine(String) function can't handle some edge case, such as some partitions are lingering, which are not visible to the corresponding AbstractStorageEngine

closeStorageEngine

public void closeStorageEngine(java.lang.String kafkaTopic)

cleanupAllStores

public void cleanupAllStores(VeniceConfigLoader configLoader)

getUserPartitions

public java.util.List<java.lang.Integer> getUserPartitions(java.lang.String kafkaTopicName)

closeAllStorageEngines

public void closeAllStorageEngines()

getStorageEngineRepository

public StorageEngineRepository getStorageEngineRepository()

getStorageEngine

public AbstractStorageEngine getStorageEngine(java.lang.String kafkaTopic)

getStoreAndUserPartitionsMapping

public java.util.Map<java.lang.String,java.util.Set<java.lang.Integer>> getStoreAndUserPartitionsMapping()

startInner

public boolean startInner()
                   throws java.lang.Exception

Specified by:

startInner in class AbstractVeniceService

Returns:

true if the service is completely started, false if it is still starting asynchronously (in this case, it is the implementer's responsibility to set AbstractVeniceService.serviceState to AbstractVeniceService.ServiceState.STARTED upon completion of the async work).

Throws:

java.lang.Exception

stopInner

public void stopInner()
               throws VeniceException

Specified by:

stopInner in class AbstractVeniceService

Throws:

VeniceException