Package com.linkedin.davinci.store

Class DelegatingStorageEngine<P extends AbstractStoragePartition>

java.lang.Object

com.linkedin.davinci.store.DelegatingStorageEngine<P>

All Implemented Interfaces:

StorageEngine<P>, Closeable, AutoCloseable

public class DelegatingStorageEngine<P extends AbstractStoragePartition>
extends Object
implements StorageEngine<P>

Constructor Summary

Constructors

Constructor	Description
DelegatingStorageEngine(StorageEngine<P> delegate)

Method Summary

Modifier and Type	Method	Description
void	addStoragePartition(StoragePartitionConfig storagePartitionConfig)
void	addStoragePartitionIfAbsent(int partitionId)
void	adjustStoragePartition(int partitionId, StoragePartitionAdjustmentTrigger mode, StoragePartitionConfig partitionConfig)	Adjust the opened storage partition according to the provided storagePartitionConfig.
void	beginBatchWrite(StoragePartitionConfig storagePartitionConfig, Map<String,String> checkpointedInfo, Optional<Supplier<byte[]>> checksumSupplier)	A lot of storage engines support efficient methods for performing large number of writes (puts/deletes) against the data source.
boolean	checkDatabaseIntegrity(int partitionId, Map<String,String> checkpointedInfo, StoragePartitionConfig storagePartitionConfig)	checks whether the current state of the database is valid during the start of ingestion.
void	clearPartitionOffset(int partitionId)	Clear the offset associated with the partitionId in the metadata partition.
void	clearStoreVersionState()	Clear the store version state in the metadata partition.
void	close()
void	closeMetadataPartition()
void	closePartition(int partitionId)
boolean	containsPartition(int partitionId)	Return true or false based on whether a given partition exists within this storage engine
void	delete(int partitionId, byte[] key)
void	deleteWithReplicationMetadata(int partitionId, byte[] key, byte[] replicationMetadata)
void	drop()	Drop the whole store
void	dropPartition(int partitionId)	Removes and returns a partition from the current store
void	dropPartition(int partitionId, boolean dropMetadataPartitionWhenEmpty)	Removes and returns a partition from the current store
void	endBatchWrite(StoragePartitionConfig storagePartitionConfig)
byte[]	get(int partitionId, byte[] key)
ByteBuffer	get(int partitionId, byte[] key, ByteBuffer valueToBePopulated)
byte[]	get(int partitionId, ByteBuffer keyBuffer)
void	getByKeyPrefix(int partitionId, byte[] partialKey, BytesStreamingCallback bytesStreamingCallback)
protected StorageEngine<P>	getDelegate()	Subclasses should be able to access the delegate, but other classes should not, otherwise they could keep a handle on the delegate, while the delegate may later be swapped via setDelegate(StorageEngine).
AbstractStorageIterator	getIterator(int partitionId)
Set<Integer>	getPartitionIds()	Get all Partition Ids which are assigned to the current Node.
Optional<OffsetRecord>	getPartitionOffset(int partitionId, PubSubContext pubSubContext)	Retrieve the offset associated with the partitionId from the metadata partition.
P	getPartitionOrThrow(int partitionId)
Set<Integer>	getPersistedPartitionIds()
byte[]	getReplicationMetadata(int partitionId, ByteBuffer key)
StorageEngineStats	getStats()
String	getStoreVersionName()
StoreVersionState	getStoreVersionState()	Retrieve the store version state from the metadata partition.
PersistenceType	getType()
boolean	isClosed()
boolean	isKeyUrnCompressionEnabled(int partitionId)
void	put(int partitionId, byte[] key, byte[] value)
void	put(int partitionId, byte[] key, ByteBuffer value)
void	putPartitionOffset(int partitionId, OffsetRecord offsetRecord)	Put the offset associated with the partitionId into the metadata partition.
void	putReplicationMetadata(int partitionId, byte[] key, byte[] replicationMetadata)
void	putStoreVersionState(StoreVersionState versionState)	Put the store version state into the metadata partition.
void	putWithReplicationMetadata(int partitionId, byte[] key, ByteBuffer value, byte[] replicationMetadata)
void	reopenStoragePartition(int partitionId)	Reopen the underlying database.
void	setDelegate(StorageEngine<P> delegate)	This setter can be used to swap the delegate which all calls get forwarded to.
void	setKeyDictCompressionFunction(Function<Integer,KeyUrnCompressor> keyDictCompressionFunction)
void	suppressLogs(boolean b)
Map<String,String>	sync(int partitionId)
void	updateStoreVersionStateCache(StoreVersionState versionState)	Used in ingestion isolation mode update the storage engine's cache in sync with the updates to the state in MainIngestionStorageMetadataService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DelegatingStorageEngine

public DelegatingStorageEngine(@Nonnull
 StorageEngine<P> delegate)

Method Details

setDelegate

public void setDelegate(@Nonnull
 StorageEngine<P> delegate)

This setter can be used to swap the delegate which all calls get forwarded to. In this way, in cases where a storage engine needs to be closed and later on a new one needs to be re-opened, this fact can be hidden from other classes needing to hold a reference to the storage engine. At the time of writing this JavaDoc, only the StorageService class is tasked with managing storage engine lifecycles in this way.

isKeyUrnCompressionEnabled

public boolean isKeyUrnCompressionEnabled(int partitionId)

setKeyDictCompressionFunction

public void setKeyDictCompressionFunction(Function<Integer,KeyUrnCompressor> keyDictCompressionFunction)

getDelegate

@Nonnull
protected StorageEngine<P> getDelegate()

Subclasses should be able to access the delegate, but other classes should not, otherwise they could keep a handle on the delegate, while the delegate may later be swapped via setDelegate(StorageEngine).

getStoreVersionName

public String getStoreVersionName()

Specified by:

getStoreVersionName in interface StorageEngine<P extends AbstractStoragePartition>

getType

public PersistenceType getType()

Specified by:

getType in interface StorageEngine<P extends AbstractStoragePartition>

getPersistedPartitionIds

public Set<Integer> getPersistedPartitionIds()

Specified by:

getPersistedPartitionIds in interface StorageEngine<P extends AbstractStoragePartition>

adjustStoragePartition

public void adjustStoragePartition(int partitionId,
 StoragePartitionAdjustmentTrigger mode,
 StoragePartitionConfig partitionConfig)

Description copied from interface: StorageEngine

Adjust the opened storage partition according to the provided storagePartitionConfig. It will throw exception if there is no opened storage partition for the given partition id. The reason to have is mainly used to ease the unit test.

Specified by:

adjustStoragePartition in interface StorageEngine<P extends AbstractStoragePartition>

addStoragePartitionIfAbsent

public void addStoragePartitionIfAbsent(int partitionId)

Specified by:

addStoragePartitionIfAbsent in interface StorageEngine<P extends AbstractStoragePartition>

addStoragePartition

public void addStoragePartition(StoragePartitionConfig storagePartitionConfig)

Specified by:

addStoragePartition in interface StorageEngine<P extends AbstractStoragePartition>

closePartition

public void closePartition(int partitionId)

Specified by:

closePartition in interface StorageEngine<P extends AbstractStoragePartition>

closeMetadataPartition

public void closeMetadataPartition()

Specified by:

closeMetadataPartition in interface StorageEngine<P extends AbstractStoragePartition>

dropPartition

public void dropPartition(int partitionId)

Description copied from interface: StorageEngine

Removes and returns a partition from the current store

Specified by:

dropPartition in interface StorageEngine<P extends AbstractStoragePartition>

Parameters:

partitionId - - id of partition to retrieve and remove

dropPartition

public void dropPartition(int partitionId,
 boolean dropMetadataPartitionWhenEmpty)

Description copied from interface: StorageEngine

Removes and returns a partition from the current store

Specified by:

dropPartition in interface StorageEngine<P extends AbstractStoragePartition>

Parameters:

partitionId - - id of partition to retrieve and remove

dropMetadataPartitionWhenEmpty - - if true, the whole store will be dropped if ALL partitions are removed

drop

public void drop()

Description copied from interface: StorageEngine

Drop the whole store

Specified by:

drop in interface StorageEngine<P extends AbstractStoragePartition>

sync

public Map<String,String> sync(int partitionId)

Specified by:

sync in interface StorageEngine<P extends AbstractStoragePartition>

close

public void close()
           throws VeniceException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

isClosed

public boolean isClosed()

Specified by:

isClosed in interface StorageEngine<P extends AbstractStoragePartition>

endBatchWrite

public void endBatchWrite(StoragePartitionConfig storagePartitionConfig)

Specified by:

endBatchWrite in interface StorageEngine<P extends AbstractStoragePartition>

reopenStoragePartition

public void reopenStoragePartition(int partitionId)

Description copied from interface: StorageEngine

Reopen the underlying database.

Specified by:

reopenStoragePartition in interface StorageEngine<P extends AbstractStoragePartition>

put

public void put(int partitionId,
 byte[] key,
 byte[] value)
         throws VeniceException

Specified by:

put in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

put

public void put(int partitionId,
 byte[] key,
 ByteBuffer value)
         throws VeniceException

Specified by:

put in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

putWithReplicationMetadata

public void putWithReplicationMetadata(int partitionId,
 byte[] key,
 ByteBuffer value,
 byte[] replicationMetadata)
                                throws VeniceException

Specified by:

putWithReplicationMetadata in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

putReplicationMetadata

public void putReplicationMetadata(int partitionId,
 byte[] key,
 byte[] replicationMetadata)
                            throws VeniceException

Specified by:

putReplicationMetadata in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

get

public byte[] get(int partitionId,
 byte[] key)
           throws VeniceException

Specified by:

get in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

get

public ByteBuffer get(int partitionId,
 byte[] key,
 ByteBuffer valueToBePopulated)
               throws VeniceException

Specified by:

get in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

get

public byte[] get(int partitionId,
 ByteBuffer keyBuffer)
           throws VeniceException

Specified by:

get in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

getByKeyPrefix

public void getByKeyPrefix(int partitionId,
 byte[] partialKey,
 BytesStreamingCallback bytesStreamingCallback)

Specified by:

getByKeyPrefix in interface StorageEngine<P extends AbstractStoragePartition>

delete

public void delete(int partitionId,
 byte[] key)
            throws VeniceException

Specified by:

delete in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

deleteWithReplicationMetadata

public void deleteWithReplicationMetadata(int partitionId,
 byte[] key,
 byte[] replicationMetadata)
                                   throws VeniceException

Specified by:

deleteWithReplicationMetadata in interface StorageEngine<P extends AbstractStoragePartition>

Throws:

VeniceException

getReplicationMetadata

public byte[] getReplicationMetadata(int partitionId,
 ByteBuffer key)

Specified by:

getReplicationMetadata in interface StorageEngine<P extends AbstractStoragePartition>

putPartitionOffset

public void putPartitionOffset(int partitionId,
 OffsetRecord offsetRecord)

Description copied from interface: StorageEngine

Put the offset associated with the partitionId into the metadata partition.

Specified by:

putPartitionOffset in interface StorageEngine<P extends AbstractStoragePartition>

getPartitionOffset

public Optional<OffsetRecord> getPartitionOffset(int partitionId,
 PubSubContext pubSubContext)

Description copied from interface: StorageEngine

Retrieve the offset associated with the partitionId from the metadata partition.

Specified by:

getPartitionOffset in interface StorageEngine<P extends AbstractStoragePartition>

clearPartitionOffset

public void clearPartitionOffset(int partitionId)

Description copied from interface: StorageEngine

Clear the offset associated with the partitionId in the metadata partition.

Specified by:

clearPartitionOffset in interface StorageEngine<P extends AbstractStoragePartition>

putStoreVersionState

public void putStoreVersionState(StoreVersionState versionState)

Description copied from interface: StorageEngine

Put the store version state into the metadata partition.

Specified by:

putStoreVersionState in interface StorageEngine<P extends AbstractStoragePartition>

updateStoreVersionStateCache

public void updateStoreVersionStateCache(StoreVersionState versionState)

Description copied from interface: StorageEngine

Used in ingestion isolation mode update the storage engine's cache in sync with the updates to the state in MainIngestionStorageMetadataService

Specified by:

updateStoreVersionStateCache in interface StorageEngine<P extends AbstractStoragePartition>

getStoreVersionState

public StoreVersionState getStoreVersionState()

Description copied from interface: StorageEngine

Retrieve the store version state from the metadata partition.

Specified by:

getStoreVersionState in interface StorageEngine<P extends AbstractStoragePartition>

clearStoreVersionState

public void clearStoreVersionState()

Description copied from interface: StorageEngine

Clear the store version state in the metadata partition.

Specified by:

clearStoreVersionState in interface StorageEngine<P extends AbstractStoragePartition>

containsPartition

public boolean containsPartition(int partitionId)

Description copied from interface: StorageEngine

Return true or false based on whether a given partition exists within this storage engine

Specified by:

containsPartition in interface StorageEngine<P extends AbstractStoragePartition>

Parameters:

partitionId - The partition to look for

Returns:

True/False, does the partition exist on this node

getPartitionIds

public Set<Integer> getPartitionIds()

Description copied from interface: StorageEngine

Get all Partition Ids which are assigned to the current Node.

Specified by:

getPartitionIds in interface StorageEngine<P extends AbstractStoragePartition>

Returns:

partition Ids that are hosted in the current Storage Engine.

getPartitionOrThrow

public P getPartitionOrThrow(int partitionId)

Specified by:

getPartitionOrThrow in interface StorageEngine<P extends AbstractStoragePartition>

getIterator

public AbstractStorageIterator getIterator(int partitionId)

Specified by:

getIterator in interface StorageEngine<P extends AbstractStoragePartition>

suppressLogs

public void suppressLogs(boolean b)

Specified by:

suppressLogs in interface StorageEngine<P extends AbstractStoragePartition>

getStats

public StorageEngineStats getStats()

Specified by:

getStats in interface StorageEngine<P extends AbstractStoragePartition>

beginBatchWrite

public void beginBatchWrite(StoragePartitionConfig storagePartitionConfig,
 Map<String,String> checkpointedInfo,
 Optional<Supplier<byte[]>> checksumSupplier)

Description copied from interface: StorageEngine

A lot of storage engines support efficient methods for performing large number of writes (puts/deletes) against the data source. This method puts the storage engine in this batch write mode

Specified by:

beginBatchWrite in interface StorageEngine<P extends AbstractStoragePartition>

checkDatabaseIntegrity

public boolean checkDatabaseIntegrity(int partitionId,
 Map<String,String> checkpointedInfo,
 StoragePartitionConfig storagePartitionConfig)

Description copied from interface: StorageEngine

checks whether the current state of the database is valid during the start of ingestion.

Specified by:

checkDatabaseIntegrity in interface StorageEngine<P extends AbstractStoragePartition>