Package com.linkedin.davinci.store
Class DelegatingStorageEngine<P extends AbstractStoragePartition>
java.lang.Object
com.linkedin.davinci.store.DelegatingStorageEngine<P>
- All Implemented Interfaces:
StorageEngine<P>
,Closeable
,AutoCloseable
public class DelegatingStorageEngine<P extends AbstractStoragePartition>
extends Object
implements StorageEngine<P>
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionvoid
addStoragePartition
(int partitionId) void
adjustStoragePartition
(int partitionId, StoragePartitionAdjustmentTrigger mode, StoragePartitionConfig partitionConfig) Adjust the opened storage partition according to the provided storagePartitionConfig.void
beginBatchWrite
(StoragePartitionConfig storagePartitionConfig, Map<String, String> checkpointedInfo, Optional<Supplier<byte[]>> checksumSupplier) A lot of storage engines support efficient methods for performing large number of writes (puts/deletes) against the data source.boolean
checkDatabaseIntegrity
(int partitionId, Map<String, String> checkpointedInfo, StoragePartitionConfig storagePartitionConfig) checks whether the current state of the database is valid during the start of ingestion.void
clearPartitionOffset
(int partitionId) Clear the offset associated with the partitionId in the metadata partition.void
Clear the store version state in the metadata partition.void
close()
void
void
closePartition
(int partitionId) boolean
containsPartition
(int partitionId) Return true or false based on whether a given partition exists within this storage enginevoid
delete
(int partitionId, byte[] key) void
deleteWithReplicationMetadata
(int partitionId, byte[] key, byte[] replicationMetadata) void
drop()
Drop the whole storevoid
dropPartition
(int partitionId) Removes and returns a partition from the current storevoid
dropPartition
(int partitionId, boolean dropMetadataPartitionWhenEmpty) Removes and returns a partition from the current storevoid
endBatchWrite
(StoragePartitionConfig storagePartitionConfig) byte[]
get
(int partitionId, byte[] key) get
(int partitionId, byte[] key, ByteBuffer valueToBePopulated) byte[]
get
(int partitionId, ByteBuffer keyBuffer) void
getByKeyPrefix
(int partitionId, byte[] partialKey, BytesStreamingCallback bytesStreamingCallback) protected StorageEngine<P>
Subclasses should be able to access the delegate, but other classes should not, otherwise they could keep a handle on the delegate, while the delegate may later be swapped viasetDelegate(StorageEngine)
.getIterator
(int partitionId) Get all Partition Ids which are assigned to the current Node.getPartitionOffset
(int partitionId) Retrieve the offset associated with the partitionId from the metadata partition.getPartitionOrThrow
(int partitionId) byte[]
getReplicationMetadata
(int partitionId, ByteBuffer key) getStats()
Retrieve the store version state from the metadata partition.getType()
boolean
isClosed()
void
put
(int partitionId, byte[] key, byte[] value) void
put
(int partitionId, byte[] key, ByteBuffer value) void
putPartitionOffset
(int partitionId, OffsetRecord offsetRecord) Put the offset associated with the partitionId into the metadata partition.void
putReplicationMetadata
(int partitionId, byte[] key, byte[] replicationMetadata) void
putStoreVersionState
(StoreVersionState versionState) Put the store version state into the metadata partition.void
putWithReplicationMetadata
(int partitionId, byte[] key, ByteBuffer value, byte[] replicationMetadata) void
reopenStoragePartition
(int partitionId) Reopen the underlying database.void
setDelegate
(StorageEngine<P> delegate) void
suppressLogs
(boolean b) sync
(int partitionId) void
updateStoreVersionStateCache
(StoreVersionState versionState) Used in ingestion isolation mode update the storage engine's cache in sync with the updates to the state inMainIngestionStorageMetadataService
-
Constructor Details
-
DelegatingStorageEngine
-
-
Method Details
-
setDelegate
-
getDelegate
Subclasses should be able to access the delegate, but other classes should not, otherwise they could keep a handle on the delegate, while the delegate may later be swapped viasetDelegate(StorageEngine)
. -
getStoreVersionName
- Specified by:
getStoreVersionName
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
getType
- Specified by:
getType
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
getPersistedPartitionIds
- Specified by:
getPersistedPartitionIds
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
adjustStoragePartition
public void adjustStoragePartition(int partitionId, StoragePartitionAdjustmentTrigger mode, StoragePartitionConfig partitionConfig) Description copied from interface:StorageEngine
Adjust the opened storage partition according to the provided storagePartitionConfig. It will throw exception if there is no opened storage partition for the given partition id. The reason to have is mainly used to ease the unit test.- Specified by:
adjustStoragePartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
addStoragePartition
public void addStoragePartition(int partitionId) - Specified by:
addStoragePartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
closePartition
public void closePartition(int partitionId) - Specified by:
closePartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
closeMetadataPartition
public void closeMetadataPartition()- Specified by:
closeMetadataPartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
dropPartition
public void dropPartition(int partitionId) Description copied from interface:StorageEngine
Removes and returns a partition from the current store- Specified by:
dropPartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Parameters:
partitionId
- - id of partition to retrieve and remove
-
dropPartition
public void dropPartition(int partitionId, boolean dropMetadataPartitionWhenEmpty) Description copied from interface:StorageEngine
Removes and returns a partition from the current store- Specified by:
dropPartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Parameters:
partitionId
- - id of partition to retrieve and removedropMetadataPartitionWhenEmpty
- - if true, the whole store will be dropped if ALL partitions are removed
-
drop
public void drop()Description copied from interface:StorageEngine
Drop the whole store- Specified by:
drop
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
sync
- Specified by:
sync
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
close
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Specified by:
close
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
isClosed
public boolean isClosed()- Specified by:
isClosed
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
endBatchWrite
- Specified by:
endBatchWrite
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
reopenStoragePartition
public void reopenStoragePartition(int partitionId) Description copied from interface:StorageEngine
Reopen the underlying database.- Specified by:
reopenStoragePartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
put
- Specified by:
put
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
put
- Specified by:
put
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
putWithReplicationMetadata
public void putWithReplicationMetadata(int partitionId, byte[] key, ByteBuffer value, byte[] replicationMetadata) throws VeniceException - Specified by:
putWithReplicationMetadata
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
putReplicationMetadata
public void putReplicationMetadata(int partitionId, byte[] key, byte[] replicationMetadata) throws VeniceException - Specified by:
putReplicationMetadata
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
get
- Specified by:
get
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
get
public ByteBuffer get(int partitionId, byte[] key, ByteBuffer valueToBePopulated) throws VeniceException - Specified by:
get
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
get
- Specified by:
get
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
getByKeyPrefix
public void getByKeyPrefix(int partitionId, byte[] partialKey, BytesStreamingCallback bytesStreamingCallback) - Specified by:
getByKeyPrefix
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
delete
- Specified by:
delete
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
deleteWithReplicationMetadata
public void deleteWithReplicationMetadata(int partitionId, byte[] key, byte[] replicationMetadata) throws VeniceException - Specified by:
deleteWithReplicationMetadata
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Throws:
VeniceException
-
getReplicationMetadata
- Specified by:
getReplicationMetadata
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
putPartitionOffset
Description copied from interface:StorageEngine
Put the offset associated with the partitionId into the metadata partition.- Specified by:
putPartitionOffset
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
getPartitionOffset
Description copied from interface:StorageEngine
Retrieve the offset associated with the partitionId from the metadata partition.- Specified by:
getPartitionOffset
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
clearPartitionOffset
public void clearPartitionOffset(int partitionId) Description copied from interface:StorageEngine
Clear the offset associated with the partitionId in the metadata partition.- Specified by:
clearPartitionOffset
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
putStoreVersionState
Description copied from interface:StorageEngine
Put the store version state into the metadata partition.- Specified by:
putStoreVersionState
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
updateStoreVersionStateCache
Description copied from interface:StorageEngine
Used in ingestion isolation mode update the storage engine's cache in sync with the updates to the state inMainIngestionStorageMetadataService
- Specified by:
updateStoreVersionStateCache
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
getStoreVersionState
Description copied from interface:StorageEngine
Retrieve the store version state from the metadata partition.- Specified by:
getStoreVersionState
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
clearStoreVersionState
public void clearStoreVersionState()Description copied from interface:StorageEngine
Clear the store version state in the metadata partition.- Specified by:
clearStoreVersionState
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
containsPartition
public boolean containsPartition(int partitionId) Description copied from interface:StorageEngine
Return true or false based on whether a given partition exists within this storage engine- Specified by:
containsPartition
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Parameters:
partitionId
- The partition to look for- Returns:
- True/False, does the partition exist on this node
-
getPartitionIds
Description copied from interface:StorageEngine
Get all Partition Ids which are assigned to the current Node.- Specified by:
getPartitionIds
in interfaceStorageEngine<P extends AbstractStoragePartition>
- Returns:
- partition Ids that are hosted in the current Storage Engine.
-
getPartitionOrThrow
- Specified by:
getPartitionOrThrow
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
getIterator
- Specified by:
getIterator
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
suppressLogs
public void suppressLogs(boolean b) - Specified by:
suppressLogs
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
getStats
- Specified by:
getStats
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
beginBatchWrite
public void beginBatchWrite(StoragePartitionConfig storagePartitionConfig, Map<String, String> checkpointedInfo, Optional<Supplier<byte[]>> checksumSupplier) Description copied from interface:StorageEngine
A lot of storage engines support efficient methods for performing large number of writes (puts/deletes) against the data source. This method puts the storage engine in this batch write mode- Specified by:
beginBatchWrite
in interfaceStorageEngine<P extends AbstractStoragePartition>
-
checkDatabaseIntegrity
public boolean checkDatabaseIntegrity(int partitionId, Map<String, String> checkpointedInfo, StoragePartitionConfig storagePartitionConfig) Description copied from interface:StorageEngine
checks whether the current state of the database is valid during the start of ingestion.- Specified by:
checkDatabaseIntegrity
in interfaceStorageEngine<P extends AbstractStoragePartition>
-